[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:22 dist-test-slave-t27c krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:22 dist-test-slave-t27c krb5kdc[399](info): set up 2 sockets
Sep 27 16:33:22 dist-test-slave-t27c krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 16:33:23.622810 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:23 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:23.749711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:23.861163 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:23.861521 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:23.861603 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:23.866252 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:23.866348 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:23.866381 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:23.866413 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:23.866478 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:23.872233 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:23.873318 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:23.874413 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:23.883855 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:23.883824 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:23.883899 412 server_base.cc:1034] running on GCE node
W20260927 16:33:23.883816 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:23.884671 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:23.885942 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:23.887151 412 hybrid_clock.cc:648] HybridClock initialized: now 1790526803887083 us; error 77 us; skew 500 ppm
Sep 27 16:33:23 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:23.890027 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:23.891004 412 webserver.cc:460] Webserver started at http://127.0.98.62:34841/ using document root <none> and password file <none>
I20260927 16:33:23.891598 412 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:23.891696 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:23.891935 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:23.894001 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "38bf1f4727624867aeb205edd77e90b3"
format_stamp: "Formatted at 2026-09-27 16:33:23 on dist-test-slave-t27c"
server_key: "268392945706534345a8c79d25de73aa"
server_key_iv: "f7c19a25ddb1cb589cb4a1c56d0dd028"
server_key_version: "encryptionkey@0"
I20260927 16:33:23.894656 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "38bf1f4727624867aeb205edd77e90b3"
format_stamp: "Formatted at 2026-09-27 16:33:23 on dist-test-slave-t27c"
server_key: "268392945706534345a8c79d25de73aa"
server_key_iv: "f7c19a25ddb1cb589cb4a1c56d0dd028"
server_key_version: "encryptionkey@0"
I20260927 16:33:23.898484 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:33:23.900832 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:23.902107 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:23.902276 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "38bf1f4727624867aeb205edd77e90b3"
format_stamp: "Formatted at 2026-09-27 16:33:23 on dist-test-slave-t27c"
server_key: "268392945706534345a8c79d25de73aa"
server_key_iv: "f7c19a25ddb1cb589cb4a1c56d0dd028"
server_key_version: "encryptionkey@0"
I20260927 16:33:23.902416 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:23.920603 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:23.923599 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:23.923820 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:23.930881 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38817
I20260927 16:33:23.930903 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38817 every 8 connection(s)
I20260927 16:33:23.931877 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:23.934644 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:23.936305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 412
I20260927 16:33:23.936484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:23.937759 392 external_mini_cluster.cc:1428] Setting key 0ca9b8be7d2c79696f82edb70ff45980
I20260927 16:33:23.940179 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3: Bootstrap starting.
I20260927 16:33:23.943077 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:23.944103 480 log.cc:824] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:23.946568 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3: No bootstrap required, opened a new log
I20260927 16:33:23.951174 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 16:33:23.951388 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:23.951421 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bf1f4727624867aeb205edd77e90b3, State: Initialized, Role: FOLLOWER
I20260927 16:33:23.951972 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 16:33:23.952138 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:23.952207 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:23.952302 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:23.953461 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 16:33:23.953835 480 leader_election.cc:302] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38bf1f4727624867aeb205edd77e90b3; no voters:
I20260927 16:33:23.954200 480 leader_election.cc:288] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:23.954305 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:23.954572 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [term 1 LEADER]: Becoming Leader. State: Replica: 38bf1f4727624867aeb205edd77e90b3, State: Running, Role: LEADER
I20260927 16:33:23.954878 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } }
I20260927 16:33:23.955567 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:23.956130 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38bf1f4727624867aeb205edd77e90b3. Latest consensus state: current_term: 1 leader_uuid: "38bf1f4727624867aeb205edd77e90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } } }
I20260927 16:33:23.956846 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:23.956737 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38bf1f4727624867aeb205edd77e90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bf1f4727624867aeb205edd77e90b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38817 } } }
I20260927 16:33:23.957182 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:23.957993 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 16:33:23 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:23.961447 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:23.965389 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:23.940927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.62:38817)
0927 16:33:23.941434 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:23.941454 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:23.942553 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 16:33:23.946804 (+ 4251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:23.946822 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:23.947162 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:23.947431 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:23.950466 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:23.951499 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:23.953224 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:23.953760 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:23.953877 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:23.959029 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:23.959035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:23.959051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:23.959080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:23.961202 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:23.961865 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:23.961871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:23.961876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:23.962024 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:23.962481 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:23.962484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:23.962486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:23.962974 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:23.963167 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:23.963863 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 16:33:23.964139 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":129,"threads_started":1}
I20260927 16:33:23.967420 494 catalog_manager.cc:1347] Generated new cluster ID: 1d2d128af61549678bbef8622ed1eea0
I20260927 16:33:23.967489 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:23.986747 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:23.988111 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:23.997536 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38bf1f4727624867aeb205edd77e90b3: Generated new TSK 0
I20260927 16:33:23.998878 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:24.031414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:24.146509 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:24.146895 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:24.146950 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:24.151032 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:24.151093 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:24.151153 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:24.157052 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:24.158237 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:24.159205 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:24.169247 506 server_base.cc:1034] running on GCE node
W20260927 16:33:24.169132 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:24.169143 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:24.169132 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:24.169868 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:24.170650 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:24.171864 506 hybrid_clock.cc:648] HybridClock initialized: now 1790526804171824 us; error 50 us; skew 500 ppm
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:24.175062 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:24.176337 506 webserver.cc:460] Webserver started at http://127.0.98.1:39773/ using document root <none> and password file <none>
I20260927 16:33:24.176946 506 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:24.177047 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:24.177327 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:24.180209 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "38b32c4c648f497ab5b396d3fa0229bb"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "00af8ed967598a7a948236efd5101bf4"
server_key_iv: "66229f0f204a4d0839a63af3dbda3c53"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.180955 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38b32c4c648f497ab5b396d3fa0229bb"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "00af8ed967598a7a948236efd5101bf4"
server_key_iv: "66229f0f204a4d0839a63af3dbda3c53"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.185977 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 16:33:24.188771 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.190080 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:24.190250 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "38b32c4c648f497ab5b396d3fa0229bb"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "00af8ed967598a7a948236efd5101bf4"
server_key_iv: "66229f0f204a4d0839a63af3dbda3c53"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.190392 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:24.213776 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:24.216866 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:24.217082 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:24.217676 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:24.218552 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:24.218631 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.218700 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:24.218755 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.229010 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35861
I20260927 16:33:24.229030 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35861 every 8 connection(s)
I20260927 16:33:24.230002 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:24.238325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 506
I20260927 16:33:24.238478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:24.238739 392 external_mini_cluster.cc:1428] Setting key 2a85a4f34d73a050bea81cc5ff3a31de
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:24.245253 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.231984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33259 (local address 127.0.98.62:38817)
0927 16:33:24.232141 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:24.232145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:24.233034 (+ 889us) server_negotiation.cc:407] Connection header received
0927 16:33:24.234493 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:24.234496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:24.234559 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:24.234671 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:24.235869 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.236663 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.237893 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.238150 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.238323 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.241267 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:24.241273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:24.241280 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:24.241314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:24.242965 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.243530 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.243536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.243541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.243635 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.243951 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.243955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.243959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.244215 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:24.244327 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:24.244866 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 16:33:24.245020 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:33:24.245954 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.232405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.1:33259)
0927 16:33:24.232858 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:24.232903 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:24.234276 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:24.234852 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:24.234859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:24.235226 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:24.235714 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.235727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.236835 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.236839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.237752 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.237762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.238361 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.238365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.238615 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.239336 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:24.239357 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:24.241086 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:24.243137 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.243144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.243159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.243415 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.243729 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.243731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.243733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.243857 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.244354 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:24.244363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:24.244619 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 16:33:24.244776 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 16:33:24.245039 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260927 16:33:24.247241 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 16:33:24.247474 635 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:24.247972 635 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:24.249574 444 ts_manager.cc:194] Registered new tserver with Master: 38b32c4c648f497ab5b396d3fa0229bb (127.0.98.1:35861)
I20260927 16:33:24.250836 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:24.265713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:24.376495 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:24.376853 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:24.376935 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:24.381036 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:24.381138 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:24.381229 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:24.386849 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:24.388054 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:24.389158 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:24.399201 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:24.399392 640 server_base.cc:1034] running on GCE node
W20260927 16:33:24.399201 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:24.399209 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:24.399946 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:24.400645 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:24.401798 640 hybrid_clock.cc:648] HybridClock initialized: now 1790526804401769 us; error 42 us; skew 500 ppm
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:24.404709 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:24.405601 640 webserver.cc:460] Webserver started at http://127.0.98.2:38831/ using document root <none> and password file <none>
I20260927 16:33:24.406138 640 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:24.406261 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:24.406533 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:24.408708 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "a001c7f6903944bfbe816bdab05ab349"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "48192c667e9635e139e59cfdb7de15a7"
server_key_iv: "bc939dd3988f43ce4003e37ec6d42d38"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.409291 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a001c7f6903944bfbe816bdab05ab349"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "48192c667e9635e139e59cfdb7de15a7"
server_key_iv: "bc939dd3988f43ce4003e37ec6d42d38"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.413084 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:33:24.415612 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.416793 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:24.416934 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "a001c7f6903944bfbe816bdab05ab349"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "48192c667e9635e139e59cfdb7de15a7"
server_key_iv: "bc939dd3988f43ce4003e37ec6d42d38"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.417057 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:24.437157 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:24.440090 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:24.440306 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:24.440912 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:24.442153 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:24.442253 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.442320 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:24.442373 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.452216 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41175
I20260927 16:33:24.452248 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41175 every 8 connection(s)
I20260927 16:33:24.453171 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:24.462080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 640
I20260927 16:33:24.462245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:24.462517 392 external_mini_cluster.cc:1428] Setting key 6233064c54bc1fcb13cfb6d79df43f8d
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:24.469031 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.455110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41407 (local address 127.0.98.62:38817)
0927 16:33:24.455279 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:24.455282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:24.456124 (+ 842us) server_negotiation.cc:407] Connection header received
0927 16:33:24.457603 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:24.457606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:24.457664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:24.457769 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:24.459140 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.459666 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.460846 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.461075 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.461193 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.464059 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:24.464063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:24.464066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:24.464099 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:24.465927 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.466725 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.466731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.466734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.466850 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.467714 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.467716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.467717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.467909 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:24.467995 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:24.468712 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 16:33:24.468868 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 16:33:24.470407 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.455512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.2:41407)
0927 16:33:24.455963 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:24.456006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:24.457324 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:24.457954 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:24.457961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:24.458491 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:24.458987 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.459001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.459800 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.459804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.460709 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.460718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.461235 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.461239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.461409 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.462063 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:24.462079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:24.463875 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:24.466087 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.466097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.466120 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.466588 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.466979 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.466985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.466989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.467596 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.468031 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:24.468041 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:24.468428 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 16:33:24.468662 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 16:33:24.469083 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260927 16:33:24.471760 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 16:33:24.472033 769 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:24.472568 769 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:33:24.473670 444 ts_manager.cc:194] Registered new tserver with Master: a001c7f6903944bfbe816bdab05ab349 (127.0.98.2:41175)
I20260927 16:33:24.474292 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:24.492883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:24.606904 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:24.607228 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:24.607314 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:24.611467 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:24.611532 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:24.611639 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:24.616894 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:24.617978 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:24.619041 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:24.629226 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:24.629249 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:24.629244 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:24.629355 774 server_base.cc:1034] running on GCE node
I20260927 16:33:24.629941 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:24.630718 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:24.631909 774 hybrid_clock.cc:648] HybridClock initialized: now 1790526804631890 us; error 31 us; skew 500 ppm
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:24.635238 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:24.636066 774 webserver.cc:460] Webserver started at http://127.0.98.3:44725/ using document root <none> and password file <none>
I20260927 16:33:24.636592 774 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:24.636649 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:24.636806 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:24.638928 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "3c8ceceeb649418e9503dd363097dc40"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "2cb2be541a096e49b33c92f1b5b3b138"
server_key_iv: "3879319663c830435a8e7bfc27e9b0ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.639437 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3c8ceceeb649418e9503dd363097dc40"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "2cb2be541a096e49b33c92f1b5b3b138"
server_key_iv: "3879319663c830435a8e7bfc27e9b0ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.643018 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:33:24.645349 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.646628 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:24.646757 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "3c8ceceeb649418e9503dd363097dc40"
format_stamp: "Formatted at 2026-09-27 16:33:24 on dist-test-slave-t27c"
server_key: "2cb2be541a096e49b33c92f1b5b3b138"
server_key_iv: "3879319663c830435a8e7bfc27e9b0ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:24.646855 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:24.657294 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:24.660167 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:24.660375 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:24.660953 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:24.661759 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:24.661805 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.661881 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:24.661912 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:24.672137 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40385
I20260927 16:33:24.672158 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40385 every 8 connection(s)
I20260927 16:33:24.673142 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:24.679075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 774
I20260927 16:33:24.679184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:24.679453 392 external_mini_cluster.cc:1428] Setting key 0698947e302344639916b8db9f999b12
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:24.687208 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.675103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41253 (local address 127.0.98.62:38817)
0927 16:33:24.675250 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:24.675253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:24.676131 (+ 878us) server_negotiation.cc:407] Connection header received
0927 16:33:24.677499 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:24.677503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:24.677560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:24.677666 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:24.678967 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.679616 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.680982 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.681272 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.681381 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.683863 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:24.683867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:24.683869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:24.683892 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:24.685180 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.685672 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.685675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.685676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.685754 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.686116 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.686119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.686120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.686368 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:24.686448 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:24.686932 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 16:33:24.687041 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:33:24.687973 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.675519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38817 (local address 127.0.98.3:41253)
0927 16:33:24.675970 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:24.676014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:24.677265 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:24.677856 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:24.677863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:24.678312 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:24.678807 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.678820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.679739 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.679742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.680850 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:24.680859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.681418 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.681421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:24.681563 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.682264 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:24.682280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:24.683682 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:24.685294 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.685299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.685311 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.685568 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.685837 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:24.685839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:24.685840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:24.685968 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:24.686460 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:24.686464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:24.686694 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 16:33:24.686853 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 16:33:24.687118 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260927 16:33:24.689144 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38817
I20260927 16:33:24.689395 903 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:24.689888 903 heartbeater.cc:511] Master 127.0.98.62:38817 requested a full tablet report, sending...
I20260927 16:33:24.690965 444 ts_manager.cc:194] Registered new tserver with Master: 3c8ceceeb649418e9503dd363097dc40 (127.0.98.3:40385)
I20260927 16:33:24.691484 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41253
I20260927 16:33:24.693977 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:24.720752 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.711193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.98.1:35861)
0927 16:33:24.711506 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:24.711520 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:24.711548 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:33:24.711600 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:24.711604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:24.711802 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:24.711929 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:24.712563 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.713334 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.714131 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.714596 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.714682 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.717106 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:24.717116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:24.717129 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:24.717152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:24.718900 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.719370 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.719382 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.719383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.719468 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.719761 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.719766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.719768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.720027 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:24.720257 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:24.720463 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 16:33:24.720600 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20260927 16:33:24.721864 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57744
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:24.747370 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:24.739074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.98.1:35861)
0927 16:33:24.739252 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:24.739256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:24.739272 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:24.739402 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:24.739405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:24.739450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:24.739521 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:24.740183 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.740687 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.741479 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:24.741700 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:24.741778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:24.744258 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:24.744262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:24.744264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:24.744290 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:24.745610 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.746087 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.746090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.746091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.746252 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:24.746552 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:24.746568 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:24.746572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:24.746769 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:24.746877 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:24.747135 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:33:24.747241 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 16:33:24.748498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 506
I20260927 16:33:24.754273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 640
I20260927 16:33:24.761871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 774
I20260927 16:33:24.767529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 412
2026-09-27T16:33:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2210 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[917](info): set up 2 sockets
Sep 27 16:33:24 dist-test-slave-t27c krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 16:33:25.809947 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:25 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:25.933917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:26.048928 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:26.049293 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:26.049377 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:26.053464 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:26.053594 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:26.053635 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:26.053668 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:26.053692 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:26.059219 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:26.060460 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:26.061664 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:26.071055 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.071058 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.071055 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:26.071133 930 server_base.cc:1034] running on GCE node
I20260927 16:33:26.071888 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:26.073081 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:26.074265 930 hybrid_clock.cc:648] HybridClock initialized: now 1790526806074230 us; error 48 us; skew 500 ppm
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:26.077271 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:26.078292 930 webserver.cc:460] Webserver started at http://127.0.98.62:45437/ using document root <none> and password file <none>
I20260927 16:33:26.078890 930 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:26.078989 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:26.079232 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:26.081347 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "d3c5a36b9b8d455aac9b10fd98f40b11"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "f3c0c7a32f15c9e848ab4344e6f59c4e"
server_key_iv: "3fc7878bc35d5be668d33ae7da2fa3a0"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.081931 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3c5a36b9b8d455aac9b10fd98f40b11"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "f3c0c7a32f15c9e848ab4344e6f59c4e"
server_key_iv: "3fc7878bc35d5be668d33ae7da2fa3a0"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.085834 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 16:33:26.088383 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.089730 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:26.089877 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "d3c5a36b9b8d455aac9b10fd98f40b11"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "f3c0c7a32f15c9e848ab4344e6f59c4e"
server_key_iv: "3fc7878bc35d5be668d33ae7da2fa3a0"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.090009 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:26.102135 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:26.105319 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:26.105553 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:26.112766 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44447
I20260927 16:33:26.112811 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44447 every 8 connection(s)
I20260927 16:33:26.113754 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:26.116555 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:26.119951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 930
I20260927 16:33:26.120085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:26.120348 392 external_mini_cluster.cc:1428] Setting key d9eaed89053fe3c26281696eccdfb664
I20260927 16:33:26.122509 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11: Bootstrap starting.
I20260927 16:33:26.124660 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:26.125427 998 log.cc:824] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:26.127348 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11: No bootstrap required, opened a new log
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:26.131174 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } }
I20260927 16:33:26.131405 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:26.131474 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c5a36b9b8d455aac9b10fd98f40b11, State: Initialized, Role: FOLLOWER
I20260927 16:33:26.131937 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } }
I20260927 16:33:26.132094 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:26.132182 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:26.132299 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:26.133167 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } }
I20260927 16:33:26.133529 998 leader_election.cc:302] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c5a36b9b8d455aac9b10fd98f40b11; no voters:
I20260927 16:33:26.133880 998 leader_election.cc:288] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:26.133946 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:26.134243 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [term 1 LEADER]: Becoming Leader. State: Replica: d3c5a36b9b8d455aac9b10fd98f40b11, State: Running, Role: LEADER
I20260927 16:33:26.134598 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } }
I20260927 16:33:26.135242 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.121765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.0.98.62:44447)
0927 16:33:26.122229 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.122253 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.122290 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:33:26.122984 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.123003 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.123346 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.123691 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:26.124469 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.125574 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.126506 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.126996 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.127175 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.129873 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:26.129878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:26.129891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:26.129916 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:26.131834 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.132356 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.132361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.132365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.132517 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.132806 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.132812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.132817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.133200 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:26.133651 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.133891 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.134250 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260927 16:33:26.135239 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:26.135864 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c5a36b9b8d455aac9b10fd98f40b11. Latest consensus state: current_term: 1 leader_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } } }
I20260927 16:33:26.135970 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:26.136289 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c5a36b9b8d455aac9b10fd98f40b11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44447 } } }
I20260927 16:33:26.136348 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:26.136379 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:26.139415 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:26.144847 1008 catalog_manager.cc:1347] Generated new cluster ID: d15123aa5ab44ef0b2de2a2d83e975f2
I20260927 16:33:26.144943 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:26.164690 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:26.165746 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:26.171584 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c5a36b9b8d455aac9b10fd98f40b11: Generated new TSK 0
I20260927 16:33:26.172310 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:26.213685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:26.327364 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:26.327643 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:26.327687 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:26.331974 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:26.332037 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:26.332139 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:26.337754 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:26.338891 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:26.339958 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:26.350134 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:26.350242 1024 server_base.cc:1034] running on GCE node
W20260927 16:33:26.350195 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.350134 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:26.350859 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:26.351598 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:26.352800 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790526806352782 us; error 29 us; skew 500 ppm
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:26.355756 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:26.356585 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41105/ using document root <none> and password file <none>
I20260927 16:33:26.357241 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:26.357328 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:26.357548 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:26.359781 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "c0ad10f7c497b3f08e63aca263acc97f"
server_key_iv: "0673b9c2fe4c10fe8f9a75a8780dbfc5"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.360281 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "c0ad10f7c497b3f08e63aca263acc97f"
server_key_iv: "0673b9c2fe4c10fe8f9a75a8780dbfc5"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.363984 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:33:26.366391 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.367671 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:26.367828 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "c0ad10f7c497b3f08e63aca263acc97f"
server_key_iv: "0673b9c2fe4c10fe8f9a75a8780dbfc5"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.367935 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:26.396763 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:26.399899 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:26.400089 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:26.400645 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:26.401436 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:26.401485 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.401525 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:26.401541 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.410769 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33527
I20260927 16:33:26.410784 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33527 every 8 connection(s)
I20260927 16:33:26.411710 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:26.420439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1024
I20260927 16:33:26.420584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:26.420837 392 external_mini_cluster.cc:1428] Setting key ea873addeebd99daa44986884986e355
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:26.426611 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.413639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38593 (local address 127.0.98.62:44447)
0927 16:33:26.413813 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.413817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.414696 (+ 879us) server_negotiation.cc:407] Connection header received
0927 16:33:26.416120 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.416124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.416184 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.416340 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:26.417560 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.418105 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.419304 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.419536 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.419656 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.422549 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:26.422552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:26.422555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:26.422585 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:26.424108 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.424660 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.424664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.424667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.424753 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.425202 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.425205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.425206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.425411 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:26.425498 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.426284 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.426410 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 16:33:26.427939 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.414018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44447 (local address 127.0.98.1:38593)
0927 16:33:26.414532 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.414576 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.415880 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.416521 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.416528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.416913 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:26.417405 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.417418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.418257 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.418261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.419162 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.419171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.419704 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.419708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.419894 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.420550 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:26.420567 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:26.422384 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:26.424241 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.424248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.424266 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.424545 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.424857 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.424861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.424863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.425068 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.425531 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:26.425540 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:26.425928 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.426197 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.426644 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":175,"threads_started":1}
I20260927 16:33:26.429380 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44447
I20260927 16:33:26.429641 1153 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:26.430251 1153 heartbeater.cc:511] Master 127.0.98.62:44447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:26.431834 962 ts_manager.cc:194] Registered new tserver with Master: d8d0e64151f641e9bf03c9d96cd5dfd9 (127.0.98.1:33527)
I20260927 16:33:26.433102 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:26.449723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:26.560467 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:26.560745 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:26.560799 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:26.564985 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:26.565054 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:26.565199 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:26.570623 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:26.571878 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:26.573016 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:26.582495 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.582578 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.582674 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:26.582832 1158 server_base.cc:1034] running on GCE node
I20260927 16:33:26.583339 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:26.584002 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:26.585202 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790526806585165 us; error 53 us; skew 500 ppm
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:26.588258 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:26.589241 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40159/ using document root <none> and password file <none>
I20260927 16:33:26.589846 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:26.589948 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:26.590247 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:26.592338 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c03fa9f0a69409c80554ac468bb8be4"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "612b844ed612951e7a5ee9db2b15f796"
server_key_iv: "c3c99aedb2e6f35b04a4550b6623f747"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.592964 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c03fa9f0a69409c80554ac468bb8be4"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "612b844ed612951e7a5ee9db2b15f796"
server_key_iv: "c3c99aedb2e6f35b04a4550b6623f747"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.596961 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 16:33:26.599783 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.601104 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:26.601253 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "2c03fa9f0a69409c80554ac468bb8be4"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "612b844ed612951e7a5ee9db2b15f796"
server_key_iv: "c3c99aedb2e6f35b04a4550b6623f747"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.601385 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:26.624797 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:26.628326 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:26.628567 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:26.629202 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:26.630065 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:26.630146 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.630229 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:26.630286 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.640427 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34779
I20260927 16:33:26.640453 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34779 every 8 connection(s)
I20260927 16:33:26.641389 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:26.646347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1158
I20260927 16:33:26.646463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:26.646728 392 external_mini_cluster.cc:1428] Setting key 4b01ae64fc38bf345074c3f1013fddbc
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:33:26.657250 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.643317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53451 (local address 127.0.98.62:44447)
0927 16:33:26.643453 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.643457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.644299 (+ 842us) server_negotiation.cc:407] Connection header received
0927 16:33:26.645694 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.645698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.645759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.645866 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:26.647402 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.647931 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.649703 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.650193 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.650318 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.653160 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:26.653166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:26.653170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:26.653202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:26.654910 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.655622 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.655628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.655632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.655724 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.656080 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.656085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.656097 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.656357 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:26.656466 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.656922 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.657021 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:26.658151 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.643693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44447 (local address 127.0.98.2:53451)
0927 16:33:26.644144 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.644185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.645455 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.646061 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.646074 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.646604 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:26.647246 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.647263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.648061 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.648068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.649497 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.649511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.650354 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.650357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.650497 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.651138 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:26.651154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:26.652969 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:26.655030 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.655035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.655047 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.655501 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.655840 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.655842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.655843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.655976 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.656466 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:26.656471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:26.656698 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.656865 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.657169 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:33:26.659418 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44447
I20260927 16:33:26.659682 1287 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:26.660272 1287 heartbeater.cc:511] Master 127.0.98.62:44447 requested a full tablet report, sending...
I20260927 16:33:26.661458 962 ts_manager.cc:194] Registered new tserver with Master: 2c03fa9f0a69409c80554ac468bb8be4 (127.0.98.2:34779)
I20260927 16:33:26.662029 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:26.677755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:26.792919 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:26.793215 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:26.793262 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:26.797603 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:26.797677 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:26.797784 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:26.803161 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:26.804323 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:26.805569 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:26.815616 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.815670 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:26.815577 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:26.815935 1292 server_base.cc:1034] running on GCE node
I20260927 16:33:26.816383 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:26.816990 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:26.818150 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790526806818134 us; error 29 us; skew 500 ppm
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:26.820992 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:26.821890 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37711/ using document root <none> and password file <none>
I20260927 16:33:26.822497 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:26.822563 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:26.822731 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:26.824842 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "31d3ca4e047f4fed9e710118dd9ba399"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "b794d1638a18c9b50ebb77e2d0084f45"
server_key_iv: "03e2ac5314bccb2f1b9ab4a153c93753"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.825413 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31d3ca4e047f4fed9e710118dd9ba399"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "b794d1638a18c9b50ebb77e2d0084f45"
server_key_iv: "03e2ac5314bccb2f1b9ab4a153c93753"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.829514 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:33:26.832098 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.833359 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:26.833456 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "31d3ca4e047f4fed9e710118dd9ba399"
format_stamp: "Formatted at 2026-09-27 16:33:26 on dist-test-slave-t27c"
server_key: "b794d1638a18c9b50ebb77e2d0084f45"
server_key_iv: "03e2ac5314bccb2f1b9ab4a153c93753"
server_key_version: "encryptionkey@0"
I20260927 16:33:26.833611 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:26.844484 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:26.847800 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:26.848011 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:26.848608 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:26.849452 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:26.849498 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.849570 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:26.849603 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:26.858316 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46045
I20260927 16:33:26.858356 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46045 every 8 connection(s)
I20260927 16:33:26.859268 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:26.864354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1292
I20260927 16:33:26.864530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:26.864835 392 external_mini_cluster.cc:1428] Setting key 9dbefb49a032e39f24915dc8fa22656f
Sep 27 16:33:26 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:26.873857 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.861212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54737 (local address 127.0.98.62:44447)
0927 16:33:26.861363 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.861366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.862247 (+ 881us) server_negotiation.cc:407] Connection header received
0927 16:33:26.863676 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.863679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.863745 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.863828 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:26.865239 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.865906 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.867586 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.867889 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.867995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.870541 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:26.870546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:26.870548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:26.870580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:26.871955 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.872478 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.872482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.872484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.872567 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.872876 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.872878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.872879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.873053 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:26.873126 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.873623 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.873725 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:33:26.874759 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.861612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44447 (local address 127.0.98.3:54737)
0927 16:33:26.862047 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.862091 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.863429 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.864006 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.864014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.864393 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:26.865072 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.865089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.866070 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.866074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.867468 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.867477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.868035 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.868038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.868196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.868789 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:26.868805 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:26.870384 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:26.872056 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.872062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.872075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.872344 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.872646 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:26.872648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:26.872650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:26.872785 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:26.873136 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:26.873141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:26.873403 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.873558 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.873825 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260927 16:33:26.875967 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44447
I20260927 16:33:26.876228 1421 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:26.876740 1421 heartbeater.cc:511] Master 127.0.98.62:44447 requested a full tablet report, sending...
I20260927 16:33:26.877818 962 ts_manager.cc:194] Registered new tserver with Master: 31d3ca4e047f4fed9e710118dd9ba399 (127.0.98.3:46045)
I20260927 16:33:26.878460 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54737
I20260927 16:33:26.879097 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:26.889741 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.882729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.98.62:44447)
0927 16:33:26.882857 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.882861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.883114 (+ 253us) server_negotiation.cc:407] Connection header received
0927 16:33:26.883159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.883161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.883201 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.883276 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:26.883944 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.884440 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.885246 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.885444 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.885519 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.886841 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:26.886843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:26.886845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:26.886866 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:26.888319 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.888685 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.888687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.888688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.888758 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:26.888979 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:26.888981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:26.888982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:26.889174 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:26.889291 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.889530 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.889619 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:33:26.894790 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:26.897778 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:26.913321 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.906724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33527 (local address 127.0.0.1:44180)
0927 16:33:26.907519 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.907575 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.907815 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.908632 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.908638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.908659 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:26.908810 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.908818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.910135 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.910138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.911652 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.911663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.912753 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.912757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.912860 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.912897 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.913015 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.913142 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":36,"spinlock_wait_cycles":6144,"thread_start_us":105,"threads_started":1}
I20260927 16:33:26.913357 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.906849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.0.98.1:33527)
0927 16:33:26.907247 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.907264 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.907681 (+ 417us) server_negotiation.cc:407] Connection header received
0927 16:33:26.908006 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.908012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.908271 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.908407 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:26.908955 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.910004 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.911789 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.912621 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.912754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.912907 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.913082 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.913197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":83,"threads_started":1}
I20260927 16:33:26.913908 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.907062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34779 (local address 127.0.0.1:44184)
0927 16:33:26.908025 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.908036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.908120 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.909725 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.909727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.909745 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:26.909825 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.909830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.911287 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.911292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.912297 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.912308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.913459 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.913465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.913559 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.913587 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.913666 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.913737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260927 16:33:26.914420 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.907995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.0.98.2:34779)
0927 16:33:26.909080 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.909088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.909104 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:26.909193 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.909200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.909432 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.909573 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:26.909955 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.911149 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.912423 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.913143 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.913256 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.913391 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.914136 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.914273 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":106,"threads_started":1}
I20260927 16:33:26.914789 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.906799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46045 (local address 127.0.0.1:58656)
0927 16:33:26.907663 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:26.907678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:26.907814 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:26.908900 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:26.908905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:26.908930 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:26.909091 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.909099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.910472 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.910475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.911659 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:26.911664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.914359 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.914364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:26.914468 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.914497 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:33:26.914576 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:33:26.914640 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":43,"spinlock_wait_cycles":5248,"thread_start_us":96,"threads_started":1}
I20260927 16:33:26.915273 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:26.907584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.98.3:46045)
0927 16:33:26.908402 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:26.908412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:26.908431 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:33:26.908501 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:26.908506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:26.908688 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:26.908847 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:26.909232 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.910331 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.912968 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:26.913998 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:26.914125 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:26.914296 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:26.915058 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 16:33:26.915149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":106,"threads_started":1}
I20260927 16:33:26.917354 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 67bad30509a24304bf718d2ccd990d0e (DEFAULT_TABLE table=test-table [id=4239afe23fec435284f8ca44e5b80687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:26.917356 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 67bad30509a24304bf718d2ccd990d0e (DEFAULT_TABLE table=test-table [id=4239afe23fec435284f8ca44e5b80687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:26.918533 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bad30509a24304bf718d2ccd990d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:26.918553 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bad30509a24304bf718d2ccd990d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:26.920845 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 67bad30509a24304bf718d2ccd990d0e (DEFAULT_TABLE table=test-table [id=4239afe23fec435284f8ca44e5b80687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:26.921875 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bad30509a24304bf718d2ccd990d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:26.926677 1438 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Bootstrap starting.
I20260927 16:33:26.927490 1439 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Bootstrap starting.
I20260927 16:33:26.928642 1440 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Bootstrap starting.
I20260927 16:33:26.929877 1438 tablet_bootstrap.cc:654] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:26.930148 1439 tablet_bootstrap.cc:654] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:26.930709 1438 log.cc:824] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:26.930794 1440 tablet_bootstrap.cc:654] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:26.930977 1439 log.cc:824] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:26.932435 1440 log.cc:824] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:26.932843 1439 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: No bootstrap required, opened a new log
I20260927 16:33:26.933045 1439 ts_tablet_manager.cc:1397] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260927 16:33:26.933188 1438 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: No bootstrap required, opened a new log
I20260927 16:33:26.933375 1438 ts_tablet_manager.cc:1397] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 16:33:26.934334 1440 tablet_bootstrap.cc:492] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: No bootstrap required, opened a new log
I20260927 16:33:26.934507 1440 ts_tablet_manager.cc:1397] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:33:26.937109 1439 raft_consensus.cc:348] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.937471 1439 raft_consensus.cc:374] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:26.937534 1439 raft_consensus.cc:729] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d0e64151f641e9bf03c9d96cd5dfd9, State: Initialized, Role: FOLLOWER
I20260927 16:33:26.937364 1438 raft_consensus.cc:348] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.937637 1438 raft_consensus.cc:374] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:26.937692 1438 raft_consensus.cc:729] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c03fa9f0a69409c80554ac468bb8be4, State: Initialized, Role: FOLLOWER
I20260927 16:33:26.938060 1439 consensus_queue.cc:260] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.938113 1438 consensus_queue.cc:260] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.938133 1440 raft_consensus.cc:348] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.938385 1440 raft_consensus.cc:374] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:26.938439 1440 raft_consensus.cc:729] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d3ca4e047f4fed9e710118dd9ba399, State: Initialized, Role: FOLLOWER
I20260927 16:33:26.938807 1440 consensus_queue.cc:260] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:26.939118 1438 ts_tablet_manager.cc:1428] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:33:26.939129 1439 ts_tablet_manager.cc:1428] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 16:33:26.939471 1421 heartbeater.cc:503] Master 127.0.98.62:44447 was elected leader, sending a full tablet report...
I20260927 16:33:26.939471 1153 heartbeater.cc:503] Master 127.0.98.62:44447 was elected leader, sending a full tablet report...
I20260927 16:33:26.940493 1287 heartbeater.cc:503] Master 127.0.98.62:44447 was elected leader, sending a full tablet report...
I20260927 16:33:26.940725 1440 ts_tablet_manager.cc:1428] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 16:33:27.110340 1422 tablet.cc:2369] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:27.121804 1444 raft_consensus.cc:482] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:27.122036 1444 raft_consensus.cc:504] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:27.124189 1444 leader_election.cc:288] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d3ca4e047f4fed9e710118dd9ba399 (127.0.98.3:46045), 2c03fa9f0a69409c80554ac468bb8be4 (127.0.98.2:34779)
I20260927 16:33:27.129743 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.123803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46045 (local address 127.0.98.1:57129)
0927 16:33:27.124653 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:27.124667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:27.124867 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:27.125493 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:27.125499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:27.125518 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:27.125623 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:27.125628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.126852 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.126859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:27.128284 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:27.128293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.129294 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.129297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:27.129400 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.129423 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:33:27.129496 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:33:27.129562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":398,"thread_start_us":98,"threads_started":1}
I20260927 16:33:27.129855 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.124001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57129 (local address 127.0.98.3:46045)
0927 16:33:27.124157 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:27.124164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:27.124789 (+ 625us) server_negotiation.cc:407] Connection header received
0927 16:33:27.125015 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:27.125018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:27.125202 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:27.125334 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:27.125812 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.126714 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.128418 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.129055 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.129156 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.129223 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:27.129550 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 16:33:27.129681 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:33:27.130851 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.123801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34779 (local address 127.0.98.1:50601)
0927 16:33:27.124502 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:27.124526 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:27.124663 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:27.125164 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:27.125169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:27.125204 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:27.125377 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:27.125387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.127856 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.127860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:27.129137 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:27.129147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.130448 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.130452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:27.130546 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.130571 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:33:27.130651 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:33:27.130719 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260927 16:33:27.130867 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.124175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50601 (local address 127.0.98.2:34779)
0927 16:33:27.124319 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:27.124326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:27.124633 (+ 307us) server_negotiation.cc:407] Connection header received
0927 16:33:27.124829 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:27.124834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:27.124904 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:27.124963 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:27.126515 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.127705 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.129261 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.129915 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.130026 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.130086 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:27.130693 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 16:33:27.130765 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:33:27.130900 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bad30509a24304bf718d2ccd990d0e" candidate_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d3ca4e047f4fed9e710118dd9ba399" is_pre_election: true
I20260927 16:33:27.131203 1375 raft_consensus.cc:2454] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d0e64151f641e9bf03c9d96cd5dfd9 in term 0.
I20260927 16:33:27.131589 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bad30509a24304bf718d2ccd990d0e" candidate_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c03fa9f0a69409c80554ac468bb8be4" is_pre_election: true
I20260927 16:33:27.131711 1043 leader_election.cc:302] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d3ca4e047f4fed9e710118dd9ba399, d8d0e64151f641e9bf03c9d96cd5dfd9; no voters:
I20260927 16:33:27.131873 1241 raft_consensus.cc:2454] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d0e64151f641e9bf03c9d96cd5dfd9 in term 0.
I20260927 16:33:27.131994 1444 raft_consensus.cc:2781] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:27.132063 1444 raft_consensus.cc:482] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:27.132122 1444 raft_consensus.cc:3037] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:27.133142 1444 raft_consensus.cc:504] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:27.133834 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bad30509a24304bf718d2ccd990d0e" candidate_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d3ca4e047f4fed9e710118dd9ba399"
I20260927 16:33:27.133888 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bad30509a24304bf718d2ccd990d0e" candidate_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c03fa9f0a69409c80554ac468bb8be4"
I20260927 16:33:27.133942 1375 raft_consensus.cc:3037] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:27.134004 1241 raft_consensus.cc:3037] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:27.135097 1241 raft_consensus.cc:2454] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d0e64151f641e9bf03c9d96cd5dfd9 in term 1.
I20260927 16:33:27.135334 1444 leader_election.cc:288] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [CANDIDATE]: Term 1 election: Requested vote from peers 31d3ca4e047f4fed9e710118dd9ba399 (127.0.98.3:46045), 2c03fa9f0a69409c80554ac468bb8be4 (127.0.98.2:34779)
I20260927 16:33:27.135504 1042 leader_election.cc:302] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c03fa9f0a69409c80554ac468bb8be4, d8d0e64151f641e9bf03c9d96cd5dfd9; no voters:
I20260927 16:33:27.135597 1375 raft_consensus.cc:2454] T 67bad30509a24304bf718d2ccd990d0e P 31d3ca4e047f4fed9e710118dd9ba399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d0e64151f641e9bf03c9d96cd5dfd9 in term 1.
I20260927 16:33:27.135732 1444 raft_consensus.cc:2781] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:27.135941 1444 raft_consensus.cc:686] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [term 1 LEADER]: Becoming Leader. State: Replica: d8d0e64151f641e9bf03c9d96cd5dfd9, State: Running, Role: LEADER
I20260927 16:33:27.136348 1444 consensus_queue.cc:237] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } }
I20260927 16:33:27.140112 961 catalog_manager.cc:5495] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d0e64151f641e9bf03c9d96cd5dfd9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d0e64151f641e9bf03c9d96cd5dfd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d3ca4e047f4fed9e710118dd9ba399" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c03fa9f0a69409c80554ac468bb8be4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34779 } health_report { overall_health: UNKNOWN } } }
W20260927 16:33:27.142586 1288 tablet.cc:2369] T 67bad30509a24304bf718d2ccd990d0e P 2c03fa9f0a69409c80554ac468bb8be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:33:27.163059 1154 tablet.cc:2369] T 67bad30509a24304bf718d2ccd990d0e P d8d0e64151f641e9bf03c9d96cd5dfd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:27.207537 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.198419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.98.1:33527)
0927 16:33:27.198575 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:27.198579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:27.198731 (+ 152us) server_negotiation.cc:407] Connection header received
0927 16:33:27.198873 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:27.198875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:27.198920 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:27.198997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:27.199653 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.200176 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.201077 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.201279 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.201353 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.204010 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:27.204021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:27.204037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:27.204059 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:27.205935 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.206382 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.206387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.206389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.206463 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.206673 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.206675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.206676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.206916 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:27.207079 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:27.207209 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 16:33:27.207307 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:27.235939 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.226880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.98.1:33527)
0927 16:33:27.227063 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:27.227067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:27.227110 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:33:27.227274 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:27.227276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:27.227321 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:27.227395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:27.228105 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.228618 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.229441 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.229663 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.229743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.232545 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:27.232550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:27.232553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:27.232584 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:27.233918 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.234495 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.234500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.234502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.234587 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.234892 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.234896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.234898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.235078 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:27.235245 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:27.235602 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 16:33:27.235721 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:27.262775 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:27.253854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.0.98.1:33527)
0927 16:33:27.254028 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:27.254033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:27.254203 (+ 170us) server_negotiation.cc:407] Connection header received
0927 16:33:27.254398 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:27.254406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:27.254452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:27.254600 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:27.255320 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.255840 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.256781 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:27.257003 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:27.257116 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:27.259627 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:27.259631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:27.259639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:27.259663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:27.260959 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.261465 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.261470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.261472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.261553 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:27.261868 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:27.261872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:27.261874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:27.262049 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:27.262230 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:27.262473 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 16:33:27.262601 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 16:33:27.264213 1067 tablet_service.cc:2908] Rejecting scan request for tablet 67bad30509a24304bf718d2ccd990d0e: Uninitialized: clean time has not yet been initialized
I20260927 16:33:27.266496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1024
I20260927 16:33:27.272704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1158
I20260927 16:33:27.277995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1292
I20260927 16:33:27.283483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 930
2026-09-27T16:33:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2516 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[1463](info): set up 2 sockets
Sep 27 16:33:27 dist-test-slave-t27c krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 16:33:28.342005 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:28 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:28.469612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35133
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:28.581336 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:28.581605 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:28.581645 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:28.585777 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:28.585845 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:33:28.585862 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:28.585881 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:28.585897 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:28.585913 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:28.591384 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35133
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:28.592463 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:28.593477 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:28.602221 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:28.602162 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:28.602190 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:28.602494 1476 server_base.cc:1034] running on GCE node
I20260927 16:33:28.603010 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:28.604195 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:28.605371 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790526808605349 us; error 31 us; skew 500 ppm
Sep 27 16:33:28 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:28.613590 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:28.614781 1476 webserver.cc:460] Webserver started at http://127.0.98.62:36151/ using document root <none> and password file <none>
I20260927 16:33:28.615356 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:28.615420 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:28.615665 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:28.617846 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "0c2111045a5e4d32b51f5c50d3aef7df"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "8aa4228b34a6ca5093013b46fb20060d"
server_key_iv: "eef1902995ff30abe6c068d81d6888d5"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.618450 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c2111045a5e4d32b51f5c50d3aef7df"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "8aa4228b34a6ca5093013b46fb20060d"
server_key_iv: "eef1902995ff30abe6c068d81d6888d5"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.622308 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:33:28.624739 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:28.626160 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:28.626363 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "0c2111045a5e4d32b51f5c50d3aef7df"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "8aa4228b34a6ca5093013b46fb20060d"
server_key_iv: "eef1902995ff30abe6c068d81d6888d5"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.626487 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:28.654631 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:28.657712 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:28.657933 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:28.666265 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35133
I20260927 16:33:28.666303 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35133 every 8 connection(s)
I20260927 16:33:28.667415 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:28.670523 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:28.675992 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Bootstrap starting.
I20260927 16:33:28.676755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1476
I20260927 16:33:28.676846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:28.677107 392 external_mini_cluster.cc:1428] Setting key a08e08a11e8ce07ab92b116cd10a2c27
I20260927 16:33:28.678802 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:28.679582 1554 log.cc:824] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:28.681437 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: No bootstrap required, opened a new log
I20260927 16:33:28.685170 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260927 16:33:28.685473 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:28.685544 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2111045a5e4d32b51f5c50d3aef7df, State: Initialized, Role: FOLLOWER
Sep 27 16:33:28 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:28.686043 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260927 16:33:28.686234 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:28.686316 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:28.686421 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:28.687649 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260927 16:33:28.688004 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c2111045a5e4d32b51f5c50d3aef7df; no voters:
I20260927 16:33:28.688375 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:28.688486 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:28.688709 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [term 1 LEADER]: Becoming Leader. State: Replica: 0c2111045a5e4d32b51f5c50d3aef7df, State: Running, Role: LEADER
I20260927 16:33:28.689040 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } }
I20260927 16:33:28.689736 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:28.690995 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } } }
I20260927 16:33:28.691138 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:28.691000 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c2111045a5e4d32b51f5c50d3aef7df. Latest consensus state: current_term: 1 leader_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2111045a5e4d32b51f5c50d3aef7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35133 } } }
I20260927 16:33:28.691236 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:28.691581 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:28.692224 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:28.678276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.0.98.62:35133)
0927 16:33:28.678714 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:28.678728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:28.678764 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:33:28.679436 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:28.679454 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:28.679843 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:28.680191 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:28.680982 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:28.682041 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:28.682948 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:28.683427 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:28.683603 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:28.686408 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:28.686414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:28.686433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:28.686469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:28.688484 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:28.689119 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:28.689125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:28.689131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:28.689263 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:28.689599 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:28.689602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:28.689604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:28.690114 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:28.690348 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:28.690844 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 16:33:28.691151 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260927 16:33:28.695166 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:28.700999 1568 catalog_manager.cc:1347] Generated new cluster ID: 933dc33b368249dbabc9cab2d3d00411
I20260927 16:33:28.701131 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:28.760608 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:28.761631 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:28.770222 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Generated new TSK 0
I20260927 16:33:28.771008 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:28.809449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:28.924454 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:28.924793 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:28.924896 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:28.928960 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:28.929023 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:28.929068 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:28.929101 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:28.934453 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:28.935556 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:28.936527 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:28.946697 1580 server_base.cc:1034] running on GCE node
W20260927 16:33:28.946652 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:28.946609 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:28.946637 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:28.947329 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:28.948076 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:28.949281 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790526808949254 us; error 69 us; skew 500 ppm
Sep 27 16:33:28 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:28.952344 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:28.953351 1580 webserver.cc:460] Webserver started at http://127.0.98.1:39981/ using document root <none> and password file <none>
I20260927 16:33:28.953938 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:28.953999 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:28.954296 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:28.956501 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "7e073b8edf3e49fb1b8b1327c9a06f9f"
server_key_iv: "df7c04f16c0c51985398d36b0ec01758"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.957100 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "7e073b8edf3e49fb1b8b1327c9a06f9f"
server_key_iv: "df7c04f16c0c51985398d36b0ec01758"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.960925 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 16:33:28.963493 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:28.964783 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:28.964942 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
format_stamp: "Formatted at 2026-09-27 16:33:28 on dist-test-slave-t27c"
server_key: "7e073b8edf3e49fb1b8b1327c9a06f9f"
server_key_iv: "df7c04f16c0c51985398d36b0ec01758"
server_key_version: "encryptionkey@0"
I20260927 16:33:28.965070 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:29.000247 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:29.003494 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:29.003705 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:29.005237 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:29.005318 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:29.005384 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:29.005419 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:29.016207 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34585
I20260927 16:33:29.016377 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34585 every 8 connection(s)
I20260927 16:33:29.017342 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:29.020768 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.006234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58265 (local address 127.0.98.62:35133)
0927 16:33:29.006399 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.006404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.007386 (+ 982us) server_negotiation.cc:407] Connection header received
0927 16:33:29.008717 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.008721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.008783 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.008884 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.010652 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.011198 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.013106 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.013381 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.013468 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.016705 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.016709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.016712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.016741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.018579 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.019211 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.019214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.019215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.019324 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.019659 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.019662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.019663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.019830 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.019951 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.020450 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.020594 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:33:29.021528 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.006672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.1:58265)
0927 16:33:29.007193 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.007254 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.008443 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.009115 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.009125 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.009641 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.010452 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.010478 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.011354 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.011360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.012960 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.012972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.013571 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.013577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.013791 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.014595 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.014620 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.016509 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.018703 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.018711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.018729 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.019096 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.019410 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.019413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.019415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.019581 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.019950 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.019958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.020208 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.020394 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.020707 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260927 16:33:29.022812 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260927 16:33:29.022953 1711 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:29.023417 1711 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260927 16:33:29.025147 1508 ts_manager.cc:194] Registered new tserver with Master: 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585)
I20260927 16:33:29.026590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1580
I20260927 16:33:29.026727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:29.026801 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58265
I20260927 16:33:29.027068 392 external_mini_cluster.cc:1428] Setting key 542d11a4f51463d131a1390de38a45b5
I20260927 16:33:29.034088 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.024895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.1:37197)
0927 16:33:29.025219 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.025236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.025368 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.025634 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.025637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.025846 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.025971 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.025978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.027272 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.027276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.028148 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.028160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.028828 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.028832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.028992 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.029657 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.029675 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.030142 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.032364 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.032369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.032371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.032599 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.032944 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.032948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.032950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.033045 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.033468 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.033471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.033579 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.033698 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.033861 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":22,"thread_start_us":97,"threads_started":1}
I20260927 16:33:29.034080 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.024985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37197 (local address 127.0.98.62:35133)
0927 16:33:29.025136 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.025139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.025302 (+ 163us) server_negotiation.cc:407] Connection header received
0927 16:33:29.025460 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.025462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.025502 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.025588 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.026090 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.027112 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.028278 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.028652 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.028829 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.030360 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.030367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.030373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.030404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.032236 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.032714 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.032720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.032722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.032820 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.033155 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.033158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.033160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.033355 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.033480 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.033770 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.033891 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:29.057480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:29.170214 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:29.170483 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:29.170526 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:29.174691 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:29.174752 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:29.174791 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:29.174858 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:29.180294 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:29.181387 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:29.182439 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:29.192864 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:29.192850 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:29.193174 1720 server_base.cc:1034] running on GCE node
W20260927 16:33:29.193269 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:29.193660 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:29.194430 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:29.195597 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790526809195580 us; error 29 us; skew 500 ppm
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:29.198380 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:29.199249 1720 webserver.cc:460] Webserver started at http://127.0.98.2:43885/ using document root <none> and password file <none>
I20260927 16:33:29.199812 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:29.199908 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:29.200140 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:29.202296 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "0b45ae39e2bd4f3d844724f1df33a6f3"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "a893e7e351109eccd1ba64aeb2b23eb9"
server_key_iv: "bd8f405c26c10586800650a5edb76ef5"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.202904 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0b45ae39e2bd4f3d844724f1df33a6f3"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "a893e7e351109eccd1ba64aeb2b23eb9"
server_key_iv: "bd8f405c26c10586800650a5edb76ef5"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.206705 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:29.209034 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:29.210232 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:29.210367 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "0b45ae39e2bd4f3d844724f1df33a6f3"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "a893e7e351109eccd1ba64aeb2b23eb9"
server_key_iv: "bd8f405c26c10586800650a5edb76ef5"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.210479 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:29.219576 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:29.222394 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:29.222541 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:29.224030 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:29.224107 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:29.224179 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:29.224228 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:29.235563 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43223
I20260927 16:33:29.236001 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43223 every 8 connection(s)
I20260927 16:33:29.236644 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:29.238950 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.224823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49447 (local address 127.0.98.62:35133)
0927 16:33:29.225001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.225006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.226093 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 16:33:29.227543 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.227547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.227609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.227714 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.229327 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.230310 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.231927 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.232175 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.232296 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.234961 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.234964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.234967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.234995 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.236817 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.237408 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.237412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.237413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.237503 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.237891 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.237893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.237895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.238066 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.238142 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.238739 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.238836 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 16:33:29.239799 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.225408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.2:49447)
0927 16:33:29.225901 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.225953 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.227231 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.227901 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.227912 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.228403 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.229167 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.229187 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.230469 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.230472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.231794 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.231804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.232336 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.232340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.232528 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.233173 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.233190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.234782 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.236925 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.236930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.236941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.237288 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.237624 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.237627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.237629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.237802 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.238276 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.238283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.238518 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.238691 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.238938 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":166,"threads_started":1}
I20260927 16:33:29.241106 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260927 16:33:29.241276 1851 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:29.241783 1851 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260927 16:33:29.242868 1508 ts_manager.cc:194] Registered new tserver with Master: 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2:43223)
I20260927 16:33:29.243623 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49447
I20260927 16:33:29.244099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1720
I20260927 16:33:29.244218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:29.244496 392 external_mini_cluster.cc:1428] Setting key 82b9cdc97b3ab4e6fb904e8498981493
I20260927 16:33:29.252562 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.243403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60717 (local address 127.0.98.62:35133)
0927 16:33:29.243524 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.243529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.243756 (+ 227us) server_negotiation.cc:407] Connection header received
0927 16:33:29.243926 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.243930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.243980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.244056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.244576 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.245216 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.247648 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.248000 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.248136 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.249374 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.249377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.249379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.249415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.250836 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.251365 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.251368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.251369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.251438 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.251738 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.251740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.251741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.251979 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.252057 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.252363 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.252454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:33:29.252703 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.243285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.2:60717)
0927 16:33:29.243675 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.243692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.243816 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.244115 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.244121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.244340 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.244444 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.244450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.245351 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.245354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.247496 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.247505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.248123 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.248125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.248192 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.248700 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.248719 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.249233 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.250958 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.250963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.250968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.251264 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.251533 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.251537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.251539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.251642 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.252091 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.252095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.252198 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.252314 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.252474 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:29.273486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35133
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:29.383797 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:29.384104 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:29.384183 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:29.388212 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:29.388350 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:29.388438 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:29.388551 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:29.394188 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:29.395313 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:29.396369 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:29.406401 1860 server_base.cc:1034] running on GCE node
W20260927 16:33:29.406365 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:29.406350 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:29.406406 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:29.406965 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:29.407689 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:29.408886 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790526809408876 us; error 29 us; skew 500 ppm
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:29.411592 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:29.412437 1860 webserver.cc:460] Webserver started at http://127.0.98.3:35541/ using document root <none> and password file <none>
I20260927 16:33:29.413021 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:29.413123 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:29.413406 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:29.415652 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "b141faee19d5dd519b0c4d2caa8acc47"
server_key_iv: "cb779e8cf929786aa47fec02d3e655ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.416376 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "b141faee19d5dd519b0c4d2caa8acc47"
server_key_iv: "cb779e8cf929786aa47fec02d3e655ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.420215 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:33:29.422750 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:29.424063 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 16:33:29.424207 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
format_stamp: "Formatted at 2026-09-27 16:33:29 on dist-test-slave-t27c"
server_key: "b141faee19d5dd519b0c4d2caa8acc47"
server_key_iv: "cb779e8cf929786aa47fec02d3e655ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:29.424328 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:29.448201 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:29.451236 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:29.451459 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:29.452907 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:29.452992 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:29.453059 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:29.453119 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:29.464092 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34489
I20260927 16:33:29.464186 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34489 every 8 connection(s)
I20260927 16:33:29.465098 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:29.467270 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.453825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36247 (local address 127.0.98.62:35133)
0927 16:33:29.454059 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.454067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.454846 (+ 779us) server_negotiation.cc:407] Connection header received
0927 16:33:29.455782 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.455785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.455843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.455951 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.457456 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.458000 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.459190 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.459410 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.459539 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.463419 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.463424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.463427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.463457 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.465086 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.465612 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.465619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.465621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.465716 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.466064 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.466069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.466072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.466331 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.466451 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.466984 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.467110 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 16:33:29.468110 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.454238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.3:36247)
0927 16:33:29.454676 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.454727 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.455567 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.456276 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.456285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.456717 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.457260 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.457273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.458127 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.458129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.459059 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.459067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.459595 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.459600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.459838 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.460520 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.460543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.463237 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.465213 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.465219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.465234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.465491 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.465796 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.465800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.465803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.465960 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.466447 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.466455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.466743 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.466928 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.467268 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
I20260927 16:33:29.469527 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35133
I20260927 16:33:29.469722 1991 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:29.469900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 1860
I20260927 16:33:29.470026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:29.470382 392 external_mini_cluster.cc:1428] Setting key 9b6bd0c433fff77bb126670680a0e66d
I20260927 16:33:29.470439 1991 heartbeater.cc:511] Master 127.0.98.62:35133 requested a full tablet report, sending...
I20260927 16:33:29.471552 1508 ts_manager.cc:194] Registered new tserver with Master: 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489)
I20260927 16:33:29.472327 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36247
I20260927 16:33:29.474080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:29.481132 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.472716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57829 (local address 127.0.98.62:35133)
0927 16:33:29.472812 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.472817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.472829 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:33:29.472864 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.472866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.472909 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.472990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.473493 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.474212 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.475041 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.475269 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.475383 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.476731 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.476733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.476735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.476756 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.478351 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.478836 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.478838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.478839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.478908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.480314 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.480317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.480318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.480490 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.480625 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.480880 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.480981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:33:29.481179 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.471964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.98.3:57829)
0927 16:33:29.472230 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.472245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.472338 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.473025 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.473028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.473253 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.473366 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.473373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.474324 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.474328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.474903 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.474910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.475386 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.475389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.475483 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.476075 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.476096 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.476585 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.478465 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.478470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.478473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.478733 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.479004 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.479007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.479009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.479118 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.480612 (+ 1494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.480615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.480701 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.480819 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.480962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:29.499518 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.492005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.0.98.62:35133)
0927 16:33:29.492116 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.492119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.492130 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:33:29.492161 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.492163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.492196 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.492257 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.492911 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.493397 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.494230 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.494434 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.494522 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.496716 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.496718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.496721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.496742 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.498149 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.498580 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.498582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.498583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.498652 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.498872 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.498874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.498875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.499024 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.499117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.499313 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.499396 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:33:29.502647 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:29.505652 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:29.521718 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.515133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34585 (local address 127.0.0.1:53068)
0927 16:33:29.515981 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.516011 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.516240 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.517130 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.517133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.517153 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:29.517336 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.517346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.518551 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.518558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.520159 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.520173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.521115 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.521231 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521268 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.521398 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.521527 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":54,"thread_start_us":80,"threads_started":1}
I20260927 16:33:29.522030 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.515164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.98.3:34489)
0927 16:33:29.515493 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.515507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.516113 (+ 606us) server_negotiation.cc:407] Connection header received
0927 16:33:29.516493 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.516501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.516735 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.516897 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:29.517401 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.518636 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.520409 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.521138 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521225 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521353 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.521777 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.521891 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":156,"threads_started":1}
I20260927 16:33:29.521916 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.514986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34489 (local address 127.0.0.1:51944)
0927 16:33:29.515951 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.516010 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.516235 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.517067 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.517072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.517094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:29.517272 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.517283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.518759 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.518762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.520291 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.520301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.521403 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.521516 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521536 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.521611 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.521678 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":48,"thread_start_us":90,"threads_started":1}
I20260927 16:33:29.522410 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.515842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.0.98.1:34585)
0927 16:33:29.516622 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.516639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.516663 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:33:29.516747 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.516753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.516969 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.517144 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:29.517482 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.518412 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.520333 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.520962 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521118 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521305 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.522118 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.522257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":85,"threads_started":1}
I20260927 16:33:29.522568 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.516594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.98.2:43223)
0927 16:33:29.516912 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.516923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.516943 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:33:29.517032 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.517037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.517205 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.517358 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:29.517604 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.518991 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.520175 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.521025 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521132 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521299 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.521648 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.521745 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":129,"threads_started":1}
I20260927 16:33:29.522042 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.515742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43223 (local address 127.0.0.1:60494)
0927 16:33:29.516200 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.516214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.516299 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.517361 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.517364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.517387 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:29.517482 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.517487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.519119 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.519122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.520033 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.520042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.521365 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.521371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.521463 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.521516 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.521614 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.521691 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260927 16:33:29.525523 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 785366fd24aa46b3aed004de47e91f01 (DEFAULT_TABLE table=test-table [id=d30293b4329e4595befead3143ff9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.525534 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 785366fd24aa46b3aed004de47e91f01 (DEFAULT_TABLE table=test-table [id=d30293b4329e4595befead3143ff9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.526147 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 785366fd24aa46b3aed004de47e91f01 (DEFAULT_TABLE table=test-table [id=d30293b4329e4595befead3143ff9ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.526579 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785366fd24aa46b3aed004de47e91f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.526660 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785366fd24aa46b3aed004de47e91f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.527009 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785366fd24aa46b3aed004de47e91f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.533286 2014 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Bootstrap starting.
I20260927 16:33:29.534745 2015 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Bootstrap starting.
I20260927 16:33:29.535467 2014 tablet_bootstrap.cc:654] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.536233 2014 log.cc:824] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:29.536413 2015 tablet_bootstrap.cc:654] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.536907 2016 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Bootstrap starting.
I20260927 16:33:29.537643 2015 log.cc:824] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:29.538372 2014 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: No bootstrap required, opened a new log
I20260927 16:33:29.538573 2014 ts_tablet_manager.cc:1397] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:33:29.538980 2016 tablet_bootstrap.cc:654] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.539672 2015 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: No bootstrap required, opened a new log
I20260927 16:33:29.539741 2016 log.cc:824] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:29.539908 2015 ts_tablet_manager.cc:1397] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:33:29.541712 2016 tablet_bootstrap.cc:492] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: No bootstrap required, opened a new log
I20260927 16:33:29.541885 2016 ts_tablet_manager.cc:1397] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:33:29.542412 2014 raft_consensus.cc:348] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.542659 2014 raft_consensus.cc:374] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.542732 2014 raft_consensus.cc:729] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e4a229cc6848edb9106f1b4bae7b9b, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.543375 2014 consensus_queue.cc:260] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.543665 2015 raft_consensus.cc:348] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.543942 2015 raft_consensus.cc:374] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.544021 2015 raft_consensus.cc:729] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fe81d9a31b4fb3aace4e37a0fe9f81, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.544435 2015 consensus_queue.cc:260] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.544608 2014 ts_tablet_manager.cc:1428] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:33:29.544991 1991 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
I20260927 16:33:29.545001 1711 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
I20260927 16:33:29.545377 2015 ts_tablet_manager.cc:1428] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:33:29.545653 2016 raft_consensus.cc:348] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.545847 2016 raft_consensus.cc:374] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.545914 2016 raft_consensus.cc:729] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.546356 2016 consensus_queue.cc:260] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.547178 2016 ts_tablet_manager.cc:1428] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:33:29.547590 1851 heartbeater.cc:503] Master 127.0.98.62:35133 was elected leader, sending a full tablet report...
W20260927 16:33:29.716847 1992 tablet.cc:2369] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:29.735330 2022 raft_consensus.cc:482] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:29.735548 2022 raft_consensus.cc:504] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.736644 2022 leader_election.cc:288] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489), 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585)
W20260927 16:33:29.737854 1852 tablet.cc:2369] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:29.740728 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.736952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34585 (local address 127.0.98.2:50681)
0927 16:33:29.737113 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.737131 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.737257 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.737547 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.737550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.737570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:29.737721 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.737729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.738739 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.738743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.739521 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.739531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.740278 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.740282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.740401 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.740419 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.740519 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.740596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 16:33:29.740728 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.737068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50681 (local address 127.0.98.1:34585)
0927 16:33:29.737202 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.737207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.737230 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:33:29.737372 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.737375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.737420 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.737531 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:29.737868 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.738622 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.739681 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.740136 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.740273 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.740339 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.740534 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.740608 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:29.741374 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785366fd24aa46b3aed004de47e91f01" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" is_pre_election: true
I20260927 16:33:29.741501 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.737133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55751 (local address 127.0.98.3:34489)
0927 16:33:29.737268 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.737272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.737757 (+ 485us) server_negotiation.cc:407] Connection header received
0927 16:33:29.737924 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.737929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.737981 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.738056 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:29.738388 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.738915 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.740419 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.740885 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.740995 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.741056 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.741298 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.741365 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:33:29.741494 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.736952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34489 (local address 127.0.98.2:55751)
0927 16:33:29.737655 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.737667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.737788 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.738106 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.738109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.738128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:29.738274 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.738279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.739041 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.739046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.740265 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.740274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.741039 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.741042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.741166 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.741191 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.741284 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.741349 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":408,"thread_start_us":104,"threads_started":1}
I20260927 16:33:29.741780 1665 raft_consensus.cc:2454] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:29.742120 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785366fd24aa46b3aed004de47e91f01" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" is_pre_election: true
I20260927 16:33:29.742395 1737 leader_election.cc:302] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 77fe81d9a31b4fb3aace4e37a0fe9f81; no voters:
I20260927 16:33:29.742436 1945 raft_consensus.cc:2454] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:29.742699 2022 raft_consensus.cc:2781] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:29.742802 2022 raft_consensus.cc:482] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:29.742864 2022 raft_consensus.cc:3037] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.743880 2022 raft_consensus.cc:504] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.744237 2022 leader_election.cc:288] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489), 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585)
I20260927 16:33:29.744598 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785366fd24aa46b3aed004de47e91f01" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
I20260927 16:33:29.744627 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785366fd24aa46b3aed004de47e91f01" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
I20260927 16:33:29.744697 1945 raft_consensus.cc:3037] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.744771 1665 raft_consensus.cc:3037] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.745796 1665 raft_consensus.cc:2454] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:29.746070 1945 raft_consensus.cc:2454] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:29.746141 1737 leader_election.cc:302] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 77fe81d9a31b4fb3aace4e37a0fe9f81; no voters:
I20260927 16:33:29.746325 2022 raft_consensus.cc:2781] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:29.746557 2022 raft_consensus.cc:686] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Running, Role: LEADER
I20260927 16:33:29.746920 2022 consensus_queue.cc:237] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } }
I20260927 16:33:29.750321 1508 catalog_manager.cc:5495] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } health_report { overall_health: HEALTHY } } }
W20260927 16:33:29.768864 1712 tablet.cc:2369] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:29.816900 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.813025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.0.98.2:43223)
0927 16:33:29.813157 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.813161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.813174 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:29.813360 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.813363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.813418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.813507 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:29.813810 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.814419 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.815402 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.815611 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.815686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.816066 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:29.816465 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:29.816545 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.816635 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.816719 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:29.822467 1665 raft_consensus.cc:1261] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Refusing update from remote peer 0b45ae39e2bd4f3d844724f1df33a6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:29.822523 1945 raft_consensus.cc:1261] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Refusing update from remote peer 0b45ae39e2bd4f3d844724f1df33a6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:29.823305 2022 consensus_queue.cc:1035] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:29.823494 2024 consensus_queue.cc:1035] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:29.832167 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333997813025775616 to 7333997812726636544. Current Snapshot: MvccSnapshot[applied={T|T < 7333997813025775616}]
I20260927 16:33:29.834877 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333997813025775616 to 7333997812726636544. Current Snapshot: MvccSnapshot[applied={T|T < 7333997813025775616}]
I20260927 16:33:29.844403 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:29.844619 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:29.847266 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Sending DeleteTablet for 3 replicas of tablet 785366fd24aa46b3aed004de47e91f01
I20260927 16:33:29.848101 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 785366fd24aa46b3aed004de47e91f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53068
I20260927 16:33:29.848140 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 785366fd24aa46b3aed004de47e91f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260927 16:33:29.848583 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 785366fd24aa46b3aed004de47e91f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60494
I20260927 16:33:29.849035 2037 tablet_replica.cc:330] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: stopping tablet replica
I20260927 16:33:29.848973 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:33:29.849363 2037 raft_consensus.cc:2229] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 16:33:29.849403 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:29.849689 2037 raft_consensus.cc:2258] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:29.850332 2039 tablet_replica.cc:330] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: stopping tablet replica
I20260927 16:33:29.850603 2039 raft_consensus.cc:2229] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:29.850953 2039 raft_consensus.cc:2258] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:29.852556 2037 ts_tablet_manager.cc:1905] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:29.854197 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 43904a864afb4847b66cef3661fd2203 (DEFAULT_TABLE table=test-table [id=67f0d057711443a39d1b8360a8bc35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.854472 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43904a864afb4847b66cef3661fd2203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.854573 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 43904a864afb4847b66cef3661fd2203 (DEFAULT_TABLE table=test-table [id=67f0d057711443a39d1b8360a8bc35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.854897 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43904a864afb4847b66cef3661fd2203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.855952 2039 ts_tablet_manager.cc:1905] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:29.857147 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 43904a864afb4847b66cef3661fd2203 (DEFAULT_TABLE table=test-table [id=67f0d057711443a39d1b8360a8bc35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:29.857328 2014 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Bootstrap starting.
I20260927 16:33:29.857411 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43904a864afb4847b66cef3661fd2203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:29.858412 2014 tablet_bootstrap.cc:654] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.859586 2014 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: No bootstrap required, opened a new log
I20260927 16:33:29.859711 2014 ts_tablet_manager.cc:1397] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:29.859789 2015 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Bootstrap starting.
I20260927 16:33:29.860173 2014 raft_consensus.cc:348] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.860304 2014 raft_consensus.cc:374] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.860345 2014 raft_consensus.cc:729] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e4a229cc6848edb9106f1b4bae7b9b, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.860487 2014 consensus_queue.cc:260] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.860777 2014 ts_tablet_manager.cc:1428] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:33:29.861225 2015 tablet_bootstrap.cc:654] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.861990 2016 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Bootstrap starting.
I20260927 16:33:29.862962 2016 tablet_bootstrap.cc:654] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:29.865314 2038 tablet_replica.cc:330] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: stopping tablet replica
I20260927 16:33:29.865706 2038 raft_consensus.cc:2229] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:29.866088 2038 raft_consensus.cc:2258] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:29.867172 2015 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: No bootstrap required, opened a new log
I20260927 16:33:29.867177 2016 tablet_bootstrap.cc:492] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: No bootstrap required, opened a new log
I20260927 16:33:29.867270 2015 ts_tablet_manager.cc:1397] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 16:33:29.867272 2016 ts_tablet_manager.cc:1397] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 16:33:29.867714 2016 raft_consensus.cc:348] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.867727 2015 raft_consensus.cc:348] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.867810 2016 raft_consensus.cc:374] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.867832 2016 raft_consensus.cc:729] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.867836 2015 raft_consensus.cc:374] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:29.867867 2015 raft_consensus.cc:729] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fe81d9a31b4fb3aace4e37a0fe9f81, State: Initialized, Role: FOLLOWER
I20260927 16:33:29.867973 2016 consensus_queue.cc:260] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.868001 2015 consensus_queue.cc:260] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.868072 2032 raft_consensus.cc:482] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:29.868126 2032 raft_consensus.cc:504] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.868264 2015 ts_tablet_manager.cc:1428] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:29.868513 2032 leader_election.cc:288] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585), 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489)
I20260927 16:33:29.869278 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43904a864afb4847b66cef3661fd2203" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" is_pre_election: true
I20260927 16:33:29.869408 1665 raft_consensus.cc:2454] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:29.869743 1737 leader_election.cc:302] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 77fe81d9a31b4fb3aace4e37a0fe9f81; no voters:
I20260927 16:33:29.869910 2032 raft_consensus.cc:2781] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:29.869951 2032 raft_consensus.cc:482] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:29.870000 2032 raft_consensus.cc:3037] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.870087 2016 ts_tablet_manager.cc:1428] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 16:33:29.870687 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43904a864afb4847b66cef3661fd2203" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" is_pre_election: true
I20260927 16:33:29.870847 1945 raft_consensus.cc:2454] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:29.871794 2038 ts_tablet_manager.cc:1905] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:29.881964 2037 ts_tablet_manager.cc:1918] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:29.881964 2039 ts_tablet_manager.cc:1918] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:29.882153 2037 log.cc:1196] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/785366fd24aa46b3aed004de47e91f01
I20260927 16:33:29.882186 2039 log.cc:1196] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/785366fd24aa46b3aed004de47e91f01
I20260927 16:33:29.882583 2037 ts_tablet_manager.cc:1939] T 785366fd24aa46b3aed004de47e91f01 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting consensus metadata
I20260927 16:33:29.882601 2039 ts_tablet_manager.cc:1939] T 785366fd24aa46b3aed004de47e91f01 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting consensus metadata
I20260927 16:33:29.882850 2032 raft_consensus.cc:504] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.885660 2032 leader_election.cc:288] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585), 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489)
I20260927 16:33:29.885890 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43904a864afb4847b66cef3661fd2203" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
I20260927 16:33:29.886034 1665 raft_consensus.cc:3037] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.885970 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43904a864afb4847b66cef3661fd2203" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
I20260927 16:33:29.886125 1945 raft_consensus.cc:3037] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:29.894728 2038 ts_tablet_manager.cc:1918] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:29.894929 2038 log.cc:1196] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/785366fd24aa46b3aed004de47e91f01
I20260927 16:33:29.895361 2038 ts_tablet_manager.cc:1939] T 785366fd24aa46b3aed004de47e91f01 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting consensus metadata
I20260927 16:33:29.895800 1492 catalog_manager.cc:4841] TS 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2:43223): tablet 785366fd24aa46b3aed004de47e91f01 (table test-table [id=d30293b4329e4595befead3143ff9ffe]) successfully deleted
I20260927 16:33:29.895836 1493 catalog_manager.cc:4841] TS 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585): tablet 785366fd24aa46b3aed004de47e91f01 (table test-table [id=d30293b4329e4595befead3143ff9ffe]) successfully deleted
I20260927 16:33:29.935235 1665 raft_consensus.cc:2454] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:29.935235 1945 raft_consensus.cc:2454] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:29.935725 1736 leader_election.cc:302] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 74e4a229cc6848edb9106f1b4bae7b9b; no voters:
I20260927 16:33:29.935989 2032 raft_consensus.cc:2781] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:29.936088 2032 raft_consensus.cc:686] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Running, Role: LEADER
I20260927 16:33:29.936237 2032 consensus_queue.cc:237] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:29.937677 1492 catalog_manager.cc:4841] TS 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489): tablet 785366fd24aa46b3aed004de47e91f01 (table test-table [id=d30293b4329e4595befead3143ff9ffe]) successfully deleted
I20260927 16:33:29.938390 1508 catalog_manager.cc:5495] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:33:29 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:29.998392 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.988144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.0.98.62:35133)
0927 16:33:29.988300 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:29.988305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:29.988526 (+ 221us) server_negotiation.cc:407] Connection header received
0927 16:33:29.988674 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:29.988677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:29.988726 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:29.988820 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:29.989785 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.990576 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.991790 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.992049 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.992141 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.995002 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:29.995005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:29.995007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:29.995027 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:29.996553 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.997076 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.997079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.997081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.997157 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:29.997439 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:29.997444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:29.997446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:29.997637 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:29.997750 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:29.998072 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 16:33:29.998246 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":67}
I20260927 16:33:29.998392 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:29.988067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35133 (local address 127.0.0.1:58912)
0927 16:33:29.988411 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:29.988434 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:29.988572 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:29.988882 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:29.988886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:29.989519 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:29.989650 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.989660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.990690 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.990694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.991675 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:29.991686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:29.992190 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:29.992194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:29.992284 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:29.992927 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:29.992943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:29.994817 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:29.996689 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.996707 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.996713 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.996954 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.997257 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:29.997260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:29.997262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:29.997347 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:29.997749 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:29.997766 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:29.997888 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 16:33:29.998014 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 16:33:29.998192 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260927 16:33:30.002835 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58912:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:33:30.003712 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:30.008834 1785 tablet_service.cc:1467] Processing CreateTablet for tablet f4f5c12b16aa4d078c62d5bedf429104 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc3dbd75d0924271b844cfde0cd125ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:30.008850 1645 tablet_service.cc:1467] Processing CreateTablet for tablet f4f5c12b16aa4d078c62d5bedf429104 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc3dbd75d0924271b844cfde0cd125ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:30.009232 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f5c12b16aa4d078c62d5bedf429104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:30.009232 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f5c12b16aa4d078c62d5bedf429104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:30.012043 2016 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: Bootstrap starting.
I20260927 16:33:30.012353 1925 tablet_service.cc:1467] Processing CreateTablet for tablet f4f5c12b16aa4d078c62d5bedf429104 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc3dbd75d0924271b844cfde0cd125ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:30.012710 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f5c12b16aa4d078c62d5bedf429104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:30.013065 2016 tablet_bootstrap.cc:654] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:30.014421 2016 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: No bootstrap required, opened a new log
I20260927 16:33:30.014505 2016 ts_tablet_manager.cc:1397] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:30.014926 2016 raft_consensus.cc:348] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.015040 2016 raft_consensus.cc:374] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:30.015069 2016 raft_consensus.cc:729] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Initialized, Role: FOLLOWER
I20260927 16:33:30.015184 2016 consensus_queue.cc:260] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.015455 2016 ts_tablet_manager.cc:1428] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:30.015408 2032 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.015558 2032 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.017009 2015 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Bootstrap starting.
I20260927 16:33:30.017971 2015 tablet_bootstrap.cc:654] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:30.019179 2015 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: No bootstrap required, opened a new log
I20260927 16:33:30.019264 2015 ts_tablet_manager.cc:1397] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:30.019632 2014 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: Bootstrap starting.
I20260927 16:33:30.021085 2014 tablet_bootstrap.cc:654] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:30.022855 2015 raft_consensus.cc:348] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.022997 2015 raft_consensus.cc:374] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:30.023028 2015 raft_consensus.cc:729] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fe81d9a31b4fb3aace4e37a0fe9f81, State: Initialized, Role: FOLLOWER
I20260927 16:33:30.023149 2015 consensus_queue.cc:260] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.023384 2015 ts_tablet_manager.cc:1428] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 16:33:30.023406 2021 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.023540 2021 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.024186 2014 tablet_bootstrap.cc:492] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: No bootstrap required, opened a new log
I20260927 16:33:30.024266 2014 ts_tablet_manager.cc:1397] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 16:33:30.024679 2014 raft_consensus.cc:348] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.024816 2014 raft_consensus.cc:374] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:30.024844 2014 raft_consensus.cc:729] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e4a229cc6848edb9106f1b4bae7b9b, State: Initialized, Role: FOLLOWER
I20260927 16:33:30.024950 2014 consensus_queue.cc:260] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.025178 2020 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.025305 2020 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.025792 2014 ts_tablet_manager.cc:1428] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:30.259295 2020 raft_consensus.cc:482] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:30.259449 2020 raft_consensus.cc:504] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.260514 2020 leader_election.cc:288] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585), 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2:43223)
I20260927 16:33:30.262720 2032 raft_consensus.cc:482] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:30.262868 2032 raft_consensus.cc:504] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.263293 2032 leader_election.cc:288] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585), 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489)
I20260927 16:33:30.263640 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" is_pre_election: true
I20260927 16:33:30.263742 1945 raft_consensus.cc:2454] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:30.264001 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" is_pre_election: true
I20260927 16:33:30.264056 1736 leader_election.cc:302] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 74e4a229cc6848edb9106f1b4bae7b9b; no voters:
I20260927 16:33:30.264132 1665 raft_consensus.cc:2454] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 0.
I20260927 16:33:30.264217 2032 raft_consensus.cc:2781] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:30.264264 2032 raft_consensus.cc:482] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:30.264305 2032 raft_consensus.cc:3037] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:30.265606 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.260919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47473 (local address 127.0.98.1:34585)
0927 16:33:30.261230 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.261236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.261512 (+ 276us) server_negotiation.cc:407] Connection header received
0927 16:33:30.261581 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.261584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.261632 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.261719 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:30.262041 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.262690 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.264333 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.264959 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.265059 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.265111 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.265382 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.265474 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":58,"threads_started":1}
I20260927 16:33:30.265599 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.260904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43999 (local address 127.0.98.2:43223)
0927 16:33:30.261062 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.261067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.261253 (+ 186us) server_negotiation.cc:407] Connection header received
0927 16:33:30.261431 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.261434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.261486 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.261561 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:30.261839 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.262616 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.264305 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.264922 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.265023 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.265097 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.265426 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.265493 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:30.265826 2032 raft_consensus.cc:504] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.265630 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.260782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34585 (local address 127.0.98.3:47473)
0927 16:33:30.261324 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:30.261335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:30.261481 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:30.261744 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:30.261746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:30.261761 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:30.261898 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.261904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.262810 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.262813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.264191 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.264199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.265115 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.265118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.265227 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.265253 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:33:30.265341 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:33:30.265430 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":169,"thread_start_us":95,"threads_started":1}
I20260927 16:33:30.266212 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.260782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43223 (local address 127.0.98.3:43999)
0927 16:33:30.261164 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:30.261181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:30.261307 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:30.261587 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:30.261590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:30.261607 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:30.261732 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.261738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.262736 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.262742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.264159 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.264170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.265194 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.265197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.265268 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.265283 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:33:30.265779 (+ 496us) client_negotiation.cc:240] Negotiation successful
0927 16:33:30.266015 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":71,"thread_start_us":72,"threads_started":1}
I20260927 16:33:30.266274 2032 leader_election.cc:288] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585), 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489)
I20260927 16:33:30.266572 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81"
I20260927 16:33:30.266713 1665 raft_consensus.cc:3037] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:30.266778 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e4a229cc6848edb9106f1b4bae7b9b"
I20260927 16:33:30.266778 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" is_pre_election: true
I20260927 16:33:30.266906 1945 raft_consensus.cc:3037] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:30.267037 1805 raft_consensus.cc:2379] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74e4a229cc6848edb9106f1b4bae7b9b in current term 1: Already voted for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in this term.
I20260927 16:33:30.267889 1665 raft_consensus.cc:2454] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:30.267937 1945 raft_consensus.cc:2454] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in term 1.
I20260927 16:33:30.268191 1737 leader_election.cc:302] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 77fe81d9a31b4fb3aace4e37a0fe9f81; no voters:
I20260927 16:33:30.268282 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f5c12b16aa4d078c62d5bedf429104" candidate_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" is_pre_election: true
I20260927 16:33:30.268332 2032 raft_consensus.cc:2781] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:30.268411 2032 raft_consensus.cc:686] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b45ae39e2bd4f3d844724f1df33a6f3, State: Running, Role: LEADER
I20260927 16:33:30.268430 1665 raft_consensus.cc:2379] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74e4a229cc6848edb9106f1b4bae7b9b in current term 1: Already voted for candidate 0b45ae39e2bd4f3d844724f1df33a6f3 in this term.
I20260927 16:33:30.268540 2032 consensus_queue.cc:237] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } }
I20260927 16:33:30.268919 1877 leader_election.cc:302] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e4a229cc6848edb9106f1b4bae7b9b; no voters: 0b45ae39e2bd4f3d844724f1df33a6f3, 77fe81d9a31b4fb3aace4e37a0fe9f81
I20260927 16:33:30.269289 2020 raft_consensus.cc:2726] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 16:33:30.269354 2022 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: TxnStatusTablet state changed. Reason: New leader 0b45ae39e2bd4f3d844724f1df33a6f3. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.269460 2022 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:30.270005 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:33:30.270200 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:33:30.270974 1508 catalog_manager.cc:5495] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } health_report { overall_health: UNKNOWN } } }
I20260927 16:33:30.314636 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.310780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43223 (local address 127.0.0.1:60502)
0927 16:33:30.310985 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:30.311004 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:30.311114 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:30.311408 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:30.311410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:30.311419 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:30.311558 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.311563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.312591 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.312595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.313297 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.313306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.313884 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.313887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.313973 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.314043 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:30.314357 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:30.314370 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 16:33:30.314432 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 16:33:30.314513 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 16:33:30.314670 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.310943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.2:43223)
0927 16:33:30.311070 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.311075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.311092 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:30.311245 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.311248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.311303 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.311385 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:30.311664 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.312441 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.313416 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.313757 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.313863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.314155 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:30.314263 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:30.314337 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.314474 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.314533 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:33:30.317523 1945 raft_consensus.cc:1261] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Refusing update from remote peer 0b45ae39e2bd4f3d844724f1df33a6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:30.317873 1665 raft_consensus.cc:1261] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Refusing update from remote peer 0b45ae39e2bd4f3d844724f1df33a6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:30.318017 2022 consensus_queue.cc:1035] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:30.318305 2032 consensus_queue.cc:1035] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:30.319535 2020 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: TxnStatusTablet state changed. Reason: New leader 0b45ae39e2bd4f3d844724f1df33a6f3. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.319656 2020 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.319563 2021 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: TxnStatusTablet state changed. Reason: New leader 0b45ae39e2bd4f3d844724f1df33a6f3. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.319710 2021 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.322047 2024 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.322160 2024 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:30.322309 2024 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.322391 2024 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 0b45ae39e2bd4f3d844724f1df33a6f3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:30.322479 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333997815056535552 to 7333997814861471744. Current Snapshot: MvccSnapshot[applied={T|T < 7333997815056535552 or (T in {7333997815056535552})}]
I20260927 16:33:30.322530 2021 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.322631 2021 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 77fe81d9a31b4fb3aace4e37a0fe9f81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.323764 2020 tablet_replica.cc:439] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "0b45ae39e2bd4f3d844724f1df33a6f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43223 } } peers { permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 } } }
I20260927 16:33:30.323877 2020 tablet_replica.cc:442] T f4f5c12b16aa4d078c62d5bedf429104 P 74e4a229cc6848edb9106f1b4bae7b9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:30.337150 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.333410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57813 (local address 127.0.98.2:43223)
0927 16:33:30.333540 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.333544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.333718 (+ 174us) server_negotiation.cc:407] Connection header received
0927 16:33:30.333837 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.333839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.333883 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.333946 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:30.334246 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.334753 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.336103 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.336311 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.336389 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.336698 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:30.336759 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:30.336854 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.336971 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.337013 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:33:30.337208 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.333345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43223 (local address 127.0.98.2:57813)
0927 16:33:30.333625 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:30.333640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:30.333752 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:30.333983 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:30.333986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:30.333994 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:30.334097 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.334103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.335016 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.335020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.335984 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:30.335995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.336423 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.336427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:30.336515 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.336565 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:30.336883 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:30.336889 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:33:30.336956 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:33:30.337035 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260927 16:33:30.344178 2022 consensus_queue.cc:1035] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e4a229cc6848edb9106f1b4bae7b9b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:30.344909 1665 raft_consensus.cc:1261] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Refusing update from remote peer 0b45ae39e2bd4f3d844724f1df33a6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:30.345322 2024 consensus_queue.cc:1035] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fe81d9a31b4fb3aace4e37a0fe9f81" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:30.386231 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:30.386427 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:30.388567 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c2111045a5e4d32b51f5c50d3aef7df: Sending DeleteTablet for 3 replicas of tablet 43904a864afb4847b66cef3661fd2203
I20260927 16:33:30.389431 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 43904a864afb4847b66cef3661fd2203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53068
I20260927 16:33:30.389482 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 43904a864afb4847b66cef3661fd2203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60494
I20260927 16:33:30.389487 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 43904a864afb4847b66cef3661fd2203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260927 16:33:30.389657 2037 tablet_replica.cc:330] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: stopping tablet replica
I20260927 16:33:30.389657 2039 tablet_replica.cc:330] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: stopping tablet replica
I20260927 16:33:30.389673 2038 tablet_replica.cc:330] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: stopping tablet replica
I20260927 16:33:30.389770 2039 raft_consensus.cc:2229] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:30.389791 2038 raft_consensus.cc:2229] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:30.389804 2037 raft_consensus.cc:2229] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:30.389918 2038 raft_consensus.cc:2258] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:30.389919 2037 raft_consensus.cc:2258] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:30.389990 2039 raft_consensus.cc:2258] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:30.390797 2039 ts_tablet_manager.cc:1905] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:30.390810 2038 ts_tablet_manager.cc:1905] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:30.390879 2037 ts_tablet_manager.cc:1905] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:30.393671 2039 ts_tablet_manager.cc:1918] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:30.393671 2037 ts_tablet_manager.cc:1918] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:30.393698 2038 ts_tablet_manager.cc:1918] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:30.393741 2037 log.cc:1196] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/43904a864afb4847b66cef3661fd2203
I20260927 16:33:30.393741 2039 log.cc:1196] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/43904a864afb4847b66cef3661fd2203
I20260927 16:33:30.393772 2038 log.cc:1196] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/43904a864afb4847b66cef3661fd2203
I20260927 16:33:30.394011 2039 ts_tablet_manager.cc:1939] T 43904a864afb4847b66cef3661fd2203 P 0b45ae39e2bd4f3d844724f1df33a6f3: Deleting consensus metadata
I20260927 16:33:30.394011 2037 ts_tablet_manager.cc:1939] T 43904a864afb4847b66cef3661fd2203 P 77fe81d9a31b4fb3aace4e37a0fe9f81: Deleting consensus metadata
I20260927 16:33:30.394037 2038 ts_tablet_manager.cc:1939] T 43904a864afb4847b66cef3661fd2203 P 74e4a229cc6848edb9106f1b4bae7b9b: Deleting consensus metadata
I20260927 16:33:30.394994 1493 catalog_manager.cc:4841] TS 77fe81d9a31b4fb3aace4e37a0fe9f81 (127.0.98.1:34585): tablet 43904a864afb4847b66cef3661fd2203 (table test-table [id=67f0d057711443a39d1b8360a8bc35eb]) successfully deleted
I20260927 16:33:30.395252 1492 catalog_manager.cc:4841] TS 0b45ae39e2bd4f3d844724f1df33a6f3 (127.0.98.2:43223): tablet 43904a864afb4847b66cef3661fd2203 (table test-table [id=67f0d057711443a39d1b8360a8bc35eb]) successfully deleted
I20260927 16:33:30.395419 1492 catalog_manager.cc:4841] TS 74e4a229cc6848edb9106f1b4bae7b9b (127.0.98.3:34489): tablet 43904a864afb4847b66cef3661fd2203 (table test-table [id=67f0d057711443a39d1b8360a8bc35eb]) successfully deleted
Sep 27 16:33:30 dist-test-slave-t27c krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:30.406206 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.394774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.1:34585)
0927 16:33:30.394883 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.394888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.395507 (+ 619us) server_negotiation.cc:407] Connection header received
0927 16:33:30.395717 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.395720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.395776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.395875 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:30.396474 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.397243 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.398061 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.398428 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.398544 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.401933 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:30.401945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:30.401965 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:30.401991 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:30.404162 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:30.404622 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:30.404638 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:30.404640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:30.404739 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:30.405128 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:30.405132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:30.405134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:30.405474 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:30.405696 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.405870 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.406000 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 16:33:30.406757 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53076
I20260927 16:33:30.416827 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:30.408695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.0.98.62:35133)
0927 16:33:30.408846 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:30.408851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:30.408890 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:33:30.408994 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:30.408997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:30.409051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:30.409142 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:30.409741 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.410613 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.411462 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:30.411818 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:30.411936 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:30.413470 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:30.413473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:30.413476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:30.413504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:30.415209 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:30.415620 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:30.415623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:30.415625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:30.415712 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:30.415984 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:30.415987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:30.415988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:30.416185 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:30.416311 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:30.416542 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 16:33:30.416652 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 16:33:30.417271 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58918
I20260927 16:33:30.418530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1580
I20260927 16:33:30.426498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1720
I20260927 16:33:30.433733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1860
I20260927 16:33:30.439333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 1476
2026-09-27T16:33:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3155 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:30 dist-test-slave-t27c krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:30 dist-test-slave-t27c krb5kdc[2074](info): set up 2 sockets
Sep 27 16:33:30 dist-test-slave-t27c krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260927 16:33:31.522044 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:31 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:31.640893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:31.750681 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:31.751026 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:31.751103 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:31.755157 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:31.755250 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:33:31.755281 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:31.755326 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:31.755362 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:31.755393 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:31.760630 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34605
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:31.761718 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:31.762745 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:31.771131 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:31.771178 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:31.771131 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:31.771497 2087 server_base.cc:1034] running on GCE node
I20260927 16:33:31.772117 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:31.773219 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:31.774400 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790526811774392 us; error 32 us; skew 500 ppm
Sep 27 16:33:31 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:31.777180 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:31.778044 2087 webserver.cc:460] Webserver started at http://127.0.98.62:45951/ using document root <none> and password file <none>
I20260927 16:33:31.778635 2087 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:31.778728 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:31.778951 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:31.780939 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "3b393ef5ba7049fb8e1ecd172b30431a"
format_stamp: "Formatted at 2026-09-27 16:33:31 on dist-test-slave-t27c"
server_key: "3bd2738dacb6228f16a4c7224049b005"
server_key_iv: "570ccba511b806d0d3a12bb2096932ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:31.781486 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b393ef5ba7049fb8e1ecd172b30431a"
format_stamp: "Formatted at 2026-09-27 16:33:31 on dist-test-slave-t27c"
server_key: "3bd2738dacb6228f16a4c7224049b005"
server_key_iv: "570ccba511b806d0d3a12bb2096932ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:31.785172 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:33:31.787724 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:31.789064 2087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:31.789219 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "3b393ef5ba7049fb8e1ecd172b30431a"
format_stamp: "Formatted at 2026-09-27 16:33:31 on dist-test-slave-t27c"
server_key: "3bd2738dacb6228f16a4c7224049b005"
server_key_iv: "570ccba511b806d0d3a12bb2096932ea"
server_key_version: "encryptionkey@0"
I20260927 16:33:31.789362 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:31.811000 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:31.814420 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:31.814618 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:31.822274 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34605
I20260927 16:33:31.822314 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34605 every 8 connection(s)
I20260927 16:33:31.823202 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:31.825886 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:31.827256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2087
I20260927 16:33:31.827389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:31.827661 392 external_mini_cluster.cc:1428] Setting key 11f859a7869c08a53c8eed086a639a2f
I20260927 16:33:31.832302 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Bootstrap starting.
I20260927 16:33:31.834998 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:31.835749 2165 log.cc:824] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:33:31 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:31.837919 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: No bootstrap required, opened a new log
I20260927 16:33:31.841487 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } }
I20260927 16:33:31.841727 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:31.841804 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b393ef5ba7049fb8e1ecd172b30431a, State: Initialized, Role: FOLLOWER
I20260927 16:33:31.842281 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:31.829030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.98.62:34605)
0927 16:33:31.829450 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:31.829465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:31.829502 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:33:31.830129 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:31.830145 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:31.830561 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:31.830850 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:31.831762 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:31.832770 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:31.833911 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:31.834447 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:31.834565 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:31.837143 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:31.837149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:31.837166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:31.837195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:31.839384 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:31.839862 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:31.839867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:31.839872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:31.839978 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:31.840254 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:31.840256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:31.840257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:31.840628 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:31.840778 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:31.841082 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 16:33:31.841408 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
I20260927 16:33:31.842319 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } }
I20260927 16:33:31.842505 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:31.842592 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:31.842689 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:31.843564 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } }
I20260927 16:33:31.843891 2165 leader_election.cc:302] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b393ef5ba7049fb8e1ecd172b30431a; no voters:
I20260927 16:33:31.844211 2165 leader_election.cc:288] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:31.844364 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:31.844673 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [term 1 LEADER]: Becoming Leader. State: Replica: 3b393ef5ba7049fb8e1ecd172b30431a, State: Running, Role: LEADER
I20260927 16:33:31.844982 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } }
I20260927 16:33:31.845516 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:31.846638 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } } }
I20260927 16:33:31.846652 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b393ef5ba7049fb8e1ecd172b30431a. Latest consensus state: current_term: 1 leader_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b393ef5ba7049fb8e1ecd172b30431a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34605 } } }
I20260927 16:33:31.846913 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:31.846913 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:31.847676 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:31.850392 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:31.855262 2179 catalog_manager.cc:1347] Generated new cluster ID: 656b513388974046a579a09a651eccd0
I20260927 16:33:31.855329 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:31.865649 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:31.866889 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:31.872323 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Generated new TSK 0
I20260927 16:33:31.873062 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:31.899067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:32.009996 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:32.010315 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:32.010393 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:32.014436 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:32.014485 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:32.014551 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:33:32.014588 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:32.014652 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:32.019819 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:32.020812 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:32.021795 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:32.032933 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.032964 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.032954 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:32.033320 2191 server_base.cc:1034] running on GCE node
I20260927 16:33:32.033747 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:32.034523 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:32.035706 2191 hybrid_clock.cc:648] HybridClock initialized: now 1790526812035657 us; error 63 us; skew 500 ppm
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:32.038425 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:32.039300 2191 webserver.cc:460] Webserver started at http://127.0.98.1:33327/ using document root <none> and password file <none>
I20260927 16:33:32.039873 2191 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:32.039963 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:32.040189 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:32.042266 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "a0a5b4ad778e3d85bbd518254c78db20"
server_key_iv: "08bf5b5a0ae7fb7d97979f429572e50f"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.042834 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "a0a5b4ad778e3d85bbd518254c78db20"
server_key_iv: "08bf5b5a0ae7fb7d97979f429572e50f"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.046548 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:33:32.049011 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.050151 2191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:32.050343 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "a0a5b4ad778e3d85bbd518254c78db20"
server_key_iv: "08bf5b5a0ae7fb7d97979f429572e50f"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.050457 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:32.074447 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:32.077419 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:32.077633 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:32.079173 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:32.079252 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.079347 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:32.079402 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:32.090412 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41593
I20260927 16:33:32.090463 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41593 every 8 connection(s)
I20260927 16:33:32.091305 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:32.093515 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.079963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58081 (local address 127.0.98.62:34605)
0927 16:33:32.080101 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.080105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.081115 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 16:33:32.082160 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.082164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.082246 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.082341 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.083861 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.084831 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.086212 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.086645 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.086772 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.089715 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.089718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.089720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.089744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.091467 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.092085 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.092088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.092090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.092181 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.092500 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.092503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.092506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.092681 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.092750 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.093265 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.093376 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:32.094350 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.080495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.1:58081)
0927 16:33:32.080969 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.081012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.081947 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.082521 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.082530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.082965 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.083673 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.083695 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.084976 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.084980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.086072 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.086082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.086827 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.086833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.087054 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.087706 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.087723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.089551 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.091568 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.091575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.091591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.091957 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.092292 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.092299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.092301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.092433 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.092775 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.092779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.092993 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.093184 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.093444 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260927 16:33:32.095222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2191
I20260927 16:33:32.095358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:32.095647 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34605
I20260927 16:33:32.095686 392 external_mini_cluster.cc:1428] Setting key 8a8f9e875da417af91ff320f6652f10a
I20260927 16:33:32.095808 2322 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:32.096347 2322 heartbeater.cc:511] Master 127.0.98.62:34605 requested a full tablet report, sending...
I20260927 16:33:32.097826 2118 ts_manager.cc:194] Registered new tserver with Master: ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593)
I20260927 16:33:32.099396 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58081
I20260927 16:33:32.106827 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.098070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37679 (local address 127.0.98.62:34605)
0927 16:33:32.098166 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.098187 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.098424 (+ 237us) server_negotiation.cc:407] Connection header received
0927 16:33:32.098510 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.098515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.098570 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.098654 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.099193 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.100024 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.101901 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.102161 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.102306 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.103603 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.103606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.103608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.103628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.105065 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.105550 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.105552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.105553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.105621 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.105906 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.105908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.105910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.106056 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.106132 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.106614 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.106708 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":39}
I20260927 16:33:32.106956 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.098010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.1:37679)
0927 16:33:32.098326 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.098341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.098433 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.098687 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.098690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.098910 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.099023 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.099030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.101252 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.101255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.101804 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.101811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.102329 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.102336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.102475 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.102957 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.102976 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.103478 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.105176 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.105180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.105182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.105451 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.105711 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.105714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.105716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.105814 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.106210 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.106214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.106409 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.106548 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.106706 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:32.125310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:32.241452 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:32.241739 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:32.241835 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:32.246134 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:32.246268 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:32.246356 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:33:32.246394 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:32.246462 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:32.251986 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:32.253082 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:32.254094 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:32.263866 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.263870 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.263861 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:32.263954 2331 server_base.cc:1034] running on GCE node
I20260927 16:33:32.264480 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:32.265190 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:32.266366 2331 hybrid_clock.cc:648] HybridClock initialized: now 1790526812266348 us; error 28 us; skew 500 ppm
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:32.269047 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:32.270038 2331 webserver.cc:460] Webserver started at http://127.0.98.2:36967/ using document root <none> and password file <none>
I20260927 16:33:32.270704 2331 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:32.270810 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:32.271051 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:32.273138 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6416122765d43708c081993feca70af"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "c441c910d5ea3b212ddb6862afdad774"
server_key_iv: "81c60dcd3f4a851c1c074b25660dfe99"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.273705 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6416122765d43708c081993feca70af"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "c441c910d5ea3b212ddb6862afdad774"
server_key_iv: "81c60dcd3f4a851c1c074b25660dfe99"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.277511 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:33:32.280107 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.281347 2331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:32.281478 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "b6416122765d43708c081993feca70af"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "c441c910d5ea3b212ddb6862afdad774"
server_key_iv: "81c60dcd3f4a851c1c074b25660dfe99"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.281586 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:32.299006 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:32.301736 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:32.301940 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:32.303478 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:32.303552 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.303640 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:32.303695 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:32.314474 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45561
I20260927 16:33:32.314577 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45561 every 8 connection(s)
I20260927 16:33:32.315709 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:32.318061 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.304156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59173 (local address 127.0.98.62:34605)
0927 16:33:32.304328 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.304334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.305697 (+ 1363us) server_negotiation.cc:407] Connection header received
0927 16:33:32.307118 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.307121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.307171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.307239 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.309030 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.309730 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.311053 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.311406 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.311537 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.314218 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.314224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.314229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.314265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.315900 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.316455 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.316459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.316461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.316551 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.316907 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.316911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.316913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.317111 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.317222 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.317750 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.317893 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:33:32.318858 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.305029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.2:59173)
0927 16:33:32.305516 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.305581 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.306881 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.307471 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.307481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.308031 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.308831 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.308856 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.309865 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.309868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.310905 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.310917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.311566 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.311569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.311720 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.312382 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.312406 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.313964 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.316008 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.316013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.316025 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.316334 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.316673 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.316676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.316677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.316805 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.317219 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.317227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.317470 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.317685 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.318017 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260927 16:33:32.320113 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34605
I20260927 16:33:32.320291 2462 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:32.320771 2462 heartbeater.cc:511] Master 127.0.98.62:34605 requested a full tablet report, sending...
I20260927 16:33:32.321570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2331
I20260927 16:33:32.321697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:32.321812 2118 ts_manager.cc:194] Registered new tserver with Master: b6416122765d43708c081993feca70af (127.0.98.2:45561)
I20260927 16:33:32.322039 392 external_mini_cluster.cc:1428] Setting key ee6be33affc0110b07f1424885f0fd5e
I20260927 16:33:32.322820 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59173
I20260927 16:33:32.331466 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.322333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58449 (local address 127.0.98.62:34605)
0927 16:33:32.322532 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.322537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.322649 (+ 112us) server_negotiation.cc:407] Connection header received
0927 16:33:32.322761 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.322764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.322814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.322905 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.325235 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.325749 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.326656 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.327053 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.327168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.328308 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.328315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.328317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.328341 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.329813 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.330326 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.330329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.330330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.330400 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.330694 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.330696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.330698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.330901 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.330975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.331266 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.331353 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 16:33:32.331606 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.322225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.2:58449)
0927 16:33:32.322561 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.322573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.322685 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.322919 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.322922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.323133 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.323247 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.323253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.325889 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.325892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.326540 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.326549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.327183 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.327186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.327273 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.327791 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.327802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.328166 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.329925 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.329928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.329930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.330220 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.330493 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.330497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.330501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.330602 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.331000 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.331005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.331109 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.331219 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.331379 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:32.349704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:32.462769 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:32.463037 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:32.463112 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:32.467368 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:32.467442 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:32.467512 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:33:32.467554 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:32.467614 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:32.473188 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:32.474438 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:32.475500 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:32.484843 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.484858 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:32.484907 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:32.485137 2471 server_base.cc:1034] running on GCE node
I20260927 16:33:32.485618 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:32.486356 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:32.487532 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790526812487520 us; error 32 us; skew 500 ppm
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:32.490228 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:32.491081 2471 webserver.cc:460] Webserver started at http://127.0.98.3:43435/ using document root <none> and password file <none>
I20260927 16:33:32.491649 2471 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:32.491740 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:32.491987 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:32.494086 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "8201c2a474334f9c916e0d86a6ae45e6"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "8d7bd4fa57d3bfadd0a5991de5f9f79a"
server_key_iv: "1e0226d6ffb11aa9f54ba53e355d0903"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.494731 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8201c2a474334f9c916e0d86a6ae45e6"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "8d7bd4fa57d3bfadd0a5991de5f9f79a"
server_key_iv: "1e0226d6ffb11aa9f54ba53e355d0903"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.498452 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:32.500769 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.501952 2471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:32.502084 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "8201c2a474334f9c916e0d86a6ae45e6"
format_stamp: "Formatted at 2026-09-27 16:33:32 on dist-test-slave-t27c"
server_key: "8d7bd4fa57d3bfadd0a5991de5f9f79a"
server_key_iv: "1e0226d6ffb11aa9f54ba53e355d0903"
server_key_version: "encryptionkey@0"
I20260927 16:33:32.502226 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:32.523869 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:32.527136 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:32.527292 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:32.528640 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:32.528689 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.528766 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:32.528800 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:32.538719 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32945
I20260927 16:33:32.538784 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32945 every 8 connection(s)
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:32.539765 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:32.542918 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.529542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32917 (local address 127.0.98.62:34605)
0927 16:33:32.529664 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.529667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.530734 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 16:33:32.531862 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.531867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.531933 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.532034 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.533678 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.534623 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.536197 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.536587 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.536716 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.539455 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.539459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.539462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.539492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.541164 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.541612 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.541615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.541616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.541689 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.541974 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.541976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.541977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.542136 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.542287 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.542692 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.542798 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:33:32.543676 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.530099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.3:32917)
0927 16:33:32.530594 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.530635 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.531607 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.532323 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.532334 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.532777 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.533482 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.533506 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.534760 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.534763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.536052 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.536073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.536753 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.536760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.536971 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.537680 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.537696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.539277 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.541270 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.541275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.541285 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.541527 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.541769 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.541771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.541772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.541891 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.542282 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.542287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.542494 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.542645 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.542882 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":142,"threads_started":1}
I20260927 16:33:32.544798 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34605
I20260927 16:33:32.544951 2602 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:32.545408 2602 heartbeater.cc:511] Master 127.0.98.62:34605 requested a full tablet report, sending...
I20260927 16:33:32.545735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2471
I20260927 16:33:32.545825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:32.546209 392 external_mini_cluster.cc:1428] Setting key a751fed07df99587fa8fb337cfd3ddb0
I20260927 16:33:32.546530 2118 ts_manager.cc:194] Registered new tserver with Master: 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:32.547343 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32917
I20260927 16:33:32.548326 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:32.555186 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.547209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57443 (local address 127.0.98.62:34605)
0927 16:33:32.547459 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.547464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.547480 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:32.547622 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.547626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.547668 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.547772 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.548236 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.548984 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.550353 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.550741 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.550861 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.551966 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.551969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.551971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.551992 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.553383 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.553887 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.553889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.553890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.553961 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.554297 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.554300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.554302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.554488 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.554595 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.554929 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.555037 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260927 16:33:32.555202 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.547087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.98.3:57443)
0927 16:33:32.547405 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.547421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.547502 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.547784 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.547787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.547982 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.548133 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.548142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.549144 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.549150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.550210 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.550225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.550863 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.550866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.550937 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.551489 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.551501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.551839 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.553490 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.553498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.553501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.553792 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.554048 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.554054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.554056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.554160 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.554599 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.554603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.554705 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.554839 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.554993 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:32.575533 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.567650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.62:34605)
0927 16:33:32.567770 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.567773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.567785 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:33:32.567819 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.567821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.567856 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.567924 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.568572 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.569081 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.569859 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.570056 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.570129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.572463 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.572466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.572468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.572489 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.573710 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.574100 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.574104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.574108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.574225 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.574578 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.574583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.574586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.574857 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.575010 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.575310 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.575413 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:32.578348 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:32.581246 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:32.596218 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.589998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.98.2:45561)
0927 16:33:32.591166 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.591178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.591197 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:33:32.591271 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.591276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.591417 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.591540 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:32.591979 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.593223 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.594652 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595515 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595671 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.595898 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.595954 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.596050 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":110,"threads_started":1}
I20260927 16:33:32.596303 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.589601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.0.1:60274)
0927 16:33:32.590311 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.590325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.590418 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.591664 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.591667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.591692 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:32.591863 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.591868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.593338 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.593341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.594529 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.594539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595649 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.595787 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.595825 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.595926 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.596048 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":93,"spinlock_wait_cycles":1920,"thread_start_us":164,"threads_started":1}
I20260927 16:33:32.596354 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.590366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32945 (local address 127.0.0.1:49552)
0927 16:33:32.590678 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.590688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.590754 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.591493 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.591498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.591520 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:32.591667 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.591674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.593320 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.593325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.594890 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.594901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595948 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.596047 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.596070 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.596119 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.596183 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":133,"thread_start_us":88,"threads_started":1}
I20260927 16:33:32.596678 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.589537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.0.1:58918)
0927 16:33:32.590165 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.590242 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.590369 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.591666 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.591669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.591695 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:32.591827 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.591832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.593052 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.593058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.594914 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.594921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595940 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.596025 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.596043 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.596105 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.596165 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":240,"thread_start_us":100,"threads_started":1}
I20260927 16:33:32.597275 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.590714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.0.98.3:32945)
0927 16:33:32.591048 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.591062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.591086 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:33:32.591163 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.591167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.591342 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.591508 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:32.592007 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.593171 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595017 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595812 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.596679 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.596998 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.597045 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.597125 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
I20260927 16:33:32.597362 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.589677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.0.98.1:41593)
0927 16:33:32.591024 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.591040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.591063 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:33:32.591145 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.591151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.591361 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.591517 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:32.591947 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.592946 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595036 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.595641 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.595741 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.595880 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.596263 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.597226 (+ 963us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":93,"threads_started":1}
I20260927 16:33:32.599552 2536 tablet_service.cc:1467] Processing CreateTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 (DEFAULT_TABLE table=test-table [id=e5e34962b9504a5e90a3b9853a235e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:32.599522 2396 tablet_service.cc:1467] Processing CreateTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 (DEFAULT_TABLE table=test-table [id=e5e34962b9504a5e90a3b9853a235e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:32.599694 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 (DEFAULT_TABLE table=test-table [id=e5e34962b9504a5e90a3b9853a235e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:32.600575 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e688c250f3bb440e94c16e62b81fa3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.600575 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e688c250f3bb440e94c16e62b81fa3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.600590 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e688c250f3bb440e94c16e62b81fa3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.607337 2625 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Bootstrap starting.
I20260927 16:33:32.608551 2627 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Bootstrap starting.
I20260927 16:33:32.609658 2625 tablet_bootstrap.cc:654] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.610069 2626 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Bootstrap starting.
I20260927 16:33:32.610224 2627 tablet_bootstrap.cc:654] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.610577 2625 log.cc:824] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:32.611421 2627 log.cc:824] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:32.612098 2626 tablet_bootstrap.cc:654] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.612644 2625 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: No bootstrap required, opened a new log
I20260927 16:33:32.612843 2625 ts_tablet_manager.cc:1397] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:33:32.612861 2626 log.cc:824] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:32.613135 2627 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: No bootstrap required, opened a new log
I20260927 16:33:32.613343 2627 ts_tablet_manager.cc:1397] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 16:33:32.614912 2626 tablet_bootstrap.cc:492] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: No bootstrap required, opened a new log
I20260927 16:33:32.615092 2626 ts_tablet_manager.cc:1397] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:33:32.616356 2627 raft_consensus.cc:348] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.616591 2627 raft_consensus.cc:374] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.616645 2627 raft_consensus.cc:729] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.616740 2625 raft_consensus.cc:348] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.616993 2625 raft_consensus.cc:374] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.617057 2625 raft_consensus.cc:729] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.617122 2627 consensus_queue.cc:260] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.617472 2625 consensus_queue.cc:260] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.618103 2627 ts_tablet_manager.cc:1428] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 16:33:32.618440 2625 ts_tablet_manager.cc:1428] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:33:32.618657 2602 heartbeater.cc:503] Master 127.0.98.62:34605 was elected leader, sending a full tablet report...
I20260927 16:33:32.618798 2322 heartbeater.cc:503] Master 127.0.98.62:34605 was elected leader, sending a full tablet report...
I20260927 16:33:32.618813 2626 raft_consensus.cc:348] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.619037 2626 raft_consensus.cc:374] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.619078 2626 raft_consensus.cc:729] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6416122765d43708c081993feca70af, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.619485 2626 consensus_queue.cc:260] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.620134 2462 heartbeater.cc:503] Master 127.0.98.62:34605 was elected leader, sending a full tablet report...
I20260927 16:33:32.620398 2626 ts_tablet_manager.cc:1428] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 16:33:32.790824 2603 tablet.cc:2369] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:33:32.817113 2463 tablet.cc:2369] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:33:32.842736 2323 tablet.cc:2369] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:32.878512 2632 raft_consensus.cc:482] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:32.878768 2632 raft_consensus.cc:504] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.879926 2632 leader_election.cc:288] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), b6416122765d43708c081993feca70af (127.0.98.2:45561)
I20260927 16:33:32.883870 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.880286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49075 (local address 127.0.98.1:41593)
0927 16:33:32.880426 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.880432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.880449 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:32.880509 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.880515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.880572 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.880668 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:32.880965 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.881478 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.882923 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.883326 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.883445 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.883511 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.883675 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.883752 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:33:32.883870 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.880199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.98.3:49075)
0927 16:33:32.880315 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.880331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.880422 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.880692 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.880695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.880715 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:32.880851 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.880857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.881617 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.881623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.882808 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.882816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.883444 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.883447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.883545 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.883566 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.883662 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.883737 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 16:33:32.884413 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.880199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.3:33053)
0927 16:33:32.880628 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.880638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.880795 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.881052 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.881055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.881069 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:32.881537 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.881543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.882614 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.882617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.883356 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.883364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.883996 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.883999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.884097 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.884117 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.884185 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.884249 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":109,"spinlock_wait_cycles":19712,"thread_start_us":135,"threads_started":1}
I20260927 16:33:32.884431 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.880377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33053 (local address 127.0.98.2:45561)
0927 16:33:32.880493 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.880497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.880723 (+ 226us) server_negotiation.cc:407] Connection header received
0927 16:33:32.880915 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.880918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.880958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.881043 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:32.881710 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.882505 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.883474 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.883866 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.883989 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.884045 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.884229 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.884316 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:33:32.884743 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e688c250f3bb440e94c16e62b81fa3e9" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" is_pre_election: true
I20260927 16:33:32.885070 2276 raft_consensus.cc:2454] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 0.
I20260927 16:33:32.885047 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e688c250f3bb440e94c16e62b81fa3e9" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af" is_pre_election: true
I20260927 16:33:32.885311 2416 raft_consensus.cc:2454] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 0.
I20260927 16:33:32.885618 2490 leader_election.cc:302] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:32.885949 2632 raft_consensus.cc:2781] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:32.886034 2632 raft_consensus.cc:482] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:32.886076 2632 raft_consensus.cc:3037] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:32.887116 2632 raft_consensus.cc:504] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.887456 2632 leader_election.cc:288] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 election: Requested vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), b6416122765d43708c081993feca70af (127.0.98.2:45561)
I20260927 16:33:32.887770 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e688c250f3bb440e94c16e62b81fa3e9" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
I20260927 16:33:32.887811 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e688c250f3bb440e94c16e62b81fa3e9" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af"
I20260927 16:33:32.887882 2276 raft_consensus.cc:3037] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:32.887935 2416 raft_consensus.cc:3037] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:32.888921 2416 raft_consensus.cc:2454] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 1.
I20260927 16:33:32.888923 2276 raft_consensus.cc:2454] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 1.
I20260927 16:33:32.889219 2490 leader_election.cc:302] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:32.889381 2632 raft_consensus.cc:2781] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:32.889577 2632 raft_consensus.cc:686] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Running, Role: LEADER
I20260927 16:33:32.889889 2632 consensus_queue.cc:237] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.893025 2118 catalog_manager.cc:5495] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8201c2a474334f9c916e0d86a6ae45e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } health_report { overall_health: HEALTHY } } }
Sep 27 16:33:32 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:32.976040 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.967907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34605 (local address 127.0.0.1:58286)
0927 16:33:32.968211 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:32.968225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:32.968363 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:32.968606 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:32.968608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:32.969066 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:32.969161 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.969167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.969904 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.969910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.970489 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:32.970496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.970918 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.970921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:32.971080 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.971595 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:32.971607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:32.972975 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:32.974466 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.974477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.974484 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.974706 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.974982 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:32.974985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:32.974986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:32.975067 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:32.975439 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:32.975446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:32.975558 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 16:33:32.975672 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:33:32.975840 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":47,"thread_start_us":84,"threads_started":1}
I20260927 16:33:32.976040 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:32.967993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.98.62:34605)
0927 16:33:32.968125 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:32.968130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:32.968311 (+ 181us) server_negotiation.cc:407] Connection header received
0927 16:33:32.968464 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:32.968468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:32.968509 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:32.968589 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:32.969287 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.969800 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.970610 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:32.970814 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:32.970920 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:32.973106 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:32.973109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:32.973112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:32.973139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:32.974368 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.974812 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.974817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.974819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.974891 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:32.975159 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:32.975162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:32.975164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:32.975339 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:32.975440 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:32.975725 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 16:33:32.975912 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":48}
I20260927 16:33:32.980070 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58286:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:33:32.980820 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:32.985714 2536 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ebfad6324a63a56a77c559eed6cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e559b6500e574a30bbe10ee00743db3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:32.985595 2256 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ebfad6324a63a56a77c559eed6cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e559b6500e574a30bbe10ee00743db3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:32.985582 2396 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ebfad6324a63a56a77c559eed6cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e559b6500e574a30bbe10ee00743db3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:33:32.986387 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ebfad6324a63a56a77c559eed6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.986385 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ebfad6324a63a56a77c559eed6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.986398 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ebfad6324a63a56a77c559eed6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:32.988857 2625 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: Bootstrap starting.
I20260927 16:33:32.989323 2627 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: Bootstrap starting.
I20260927 16:33:32.989924 2625 tablet_bootstrap.cc:654] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.990365 2627 tablet_bootstrap.cc:654] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.990500 2626 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: Bootstrap starting.
I20260927 16:33:32.991384 2626 tablet_bootstrap.cc:654] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:32.991791 2627 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: No bootstrap required, opened a new log
I20260927 16:33:32.991883 2627 ts_tablet_manager.cc:1397] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 16:33:32.992347 2627 raft_consensus.cc:348] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.992481 2627 raft_consensus.cc:374] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.992517 2627 raft_consensus.cc:729] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.992650 2627 consensus_queue.cc:260] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.992784 2626 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: No bootstrap required, opened a new log
I20260927 16:33:32.992861 2626 ts_tablet_manager.cc:1397] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:32.992924 2627 ts_tablet_manager.cc:1428] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:33:32.992944 2631 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:32.993098 2631 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:32.993301 2626 raft_consensus.cc:348] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.993422 2626 raft_consensus.cc:374] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.993468 2626 raft_consensus.cc:729] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6416122765d43708c081993feca70af, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.993592 2626 consensus_queue.cc:260] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.993814 2625 tablet_bootstrap.cc:492] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: No bootstrap required, opened a new log
I20260927 16:33:32.993902 2625 ts_tablet_manager.cc:1397] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 16:33:32.993857 2633 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:32.993980 2633 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:32.994266 2626 ts_tablet_manager.cc:1428] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 16:33:32.994372 2625 raft_consensus.cc:348] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.994514 2625 raft_consensus.cc:374] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:32.994554 2625 raft_consensus.cc:729] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Initialized, Role: FOLLOWER
I20260927 16:33:32.994691 2625 consensus_queue.cc:260] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:32.994976 2625 ts_tablet_manager.cc:1428] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:32.994951 2632 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:32.995090 2632 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:33.097208 2633 raft_consensus.cc:482] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:33.097357 2633 raft_consensus.cc:504] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.098634 2633 leader_election.cc:288] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:33.102953 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.098976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39593 (local address 127.0.98.1:41593)
0927 16:33:33.099112 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.099117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.099257 (+ 140us) server_negotiation.cc:407] Connection header received
0927 16:33:33.099379 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.099383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.099432 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.099538 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:33.099822 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.100550 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.101920 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.102375 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.102501 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.102564 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.102763 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.102834 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:33.102974 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.098926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.98.2:39593)
0927 16:33:33.099150 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.099166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.099283 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.099560 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.099563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.099581 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:33.099703 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.099729 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.100664 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.100667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.101806 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.101813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.102515 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.102519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.102624 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.102644 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.102745 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.102819 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260927 16:33:33.103493 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ebfad6324a63a56a77c559eed6cf" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" is_pre_election: true
I20260927 16:33:33.103646 2276 raft_consensus.cc:2454] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 0.
I20260927 16:33:33.104015 2350 leader_election.cc:302] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:33.104279 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.099303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32945 (local address 127.0.98.2:37355)
0927 16:33:33.099721 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.099735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.099824 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.100890 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.100893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.100910 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:33.101403 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.101411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.102118 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.102124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.102911 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.102920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.103719 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.103722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.103969 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.103991 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.104072 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.104139 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":51,"threads_started":1}
I20260927 16:33:33.104331 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.099524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37355 (local address 127.0.98.3:32945)
0927 16:33:33.099737 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.099741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.100679 (+ 938us) server_negotiation.cc:407] Connection header received
0927 16:33:33.100750 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.100752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.100795 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.100861 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:33.101516 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.102002 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.103033 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.103592 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.103753 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.103830 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.104085 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.104177 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260927 16:33:33.104323 2633 raft_consensus.cc:2781] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:33.104393 2633 raft_consensus.cc:482] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:33.104435 2633 raft_consensus.cc:3037] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.104820 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ebfad6324a63a56a77c559eed6cf" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6" is_pre_election: true
I20260927 16:33:33.105042 2556 raft_consensus.cc:2454] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 0.
I20260927 16:33:33.105700 2633 raft_consensus.cc:504] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.106063 2633 leader_election.cc:288] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 election: Requested vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:33.106395 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ebfad6324a63a56a77c559eed6cf" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
I20260927 16:33:33.106513 2276 raft_consensus.cc:3037] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.106458 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ebfad6324a63a56a77c559eed6cf" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6"
I20260927 16:33:33.106601 2556 raft_consensus.cc:3037] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.107415 2276 raft_consensus.cc:2454] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 1.
I20260927 16:33:33.107494 2556 raft_consensus.cc:2454] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 1.
I20260927 16:33:33.107719 2350 leader_election.cc:302] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:33.107908 2633 raft_consensus.cc:2781] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:33.108126 2633 raft_consensus.cc:686] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [term 1 LEADER]: Becoming Leader. State: Replica: b6416122765d43708c081993feca70af, State: Running, Role: LEADER
I20260927 16:33:33.108613 2633 consensus_queue.cc:237] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.111101 2649 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: TxnStatusTablet state changed. Reason: New leader b6416122765d43708c081993feca70af. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.111268 2649 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:33.111559 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:33:33.111781 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:33:33.112471 2117 catalog_manager.cc:5495] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af reported cstate change: term changed from 0 to 1, leader changed from <none> to b6416122765d43708c081993feca70af (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20260927 16:33:33.150295 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.145678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.0.1:60280)
0927 16:33:33.145871 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.145887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.145994 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.146389 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.146391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.146398 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:33.146542 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.146547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.147686 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.147690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.148332 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.148341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.148917 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.148920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.148993 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.149062 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.149905 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.149922 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.149987 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.150095 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 16:33:33.150279 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.145802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.0.98.2:45561)
0927 16:33:33.145944 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.145949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.145966 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:33.146196 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.146200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.146268 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.146374 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.146663 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.147552 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.148451 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.148800 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.148907 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.149205 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.149797 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.149912 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.150019 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.150081 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:33.156122 2556 raft_consensus.cc:1261] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Refusing update from remote peer b6416122765d43708c081993feca70af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.156136 2276 raft_consensus.cc:1261] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Refusing update from remote peer b6416122765d43708c081993feca70af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.156780 2649 consensus_queue.cc:1035] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.156989 2633 consensus_queue.cc:1035] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.160223 2632 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: TxnStatusTablet state changed. Reason: New leader b6416122765d43708c081993feca70af. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.160360 2632 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:33.160815 2631 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: TxnStatusTablet state changed. Reason: New leader b6416122765d43708c081993feca70af. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.160928 2631 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:33.164140 2633 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.164302 2633 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:33.164589 2632 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.164711 2632 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P 8201c2a474334f9c916e0d86a6ae45e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:33.165351 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333997826679951360 to 7333997826495827968. Current Snapshot: MvccSnapshot[applied={T|T < 7333997826679951360}]
I20260927 16:33:33.166836 2649 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.166942 2649 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P b6416122765d43708c081993feca70af: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:33:33.167894 2631 tablet_replica.cc:439] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } } }
I20260927 16:33:33.168006 2631 tablet_replica.cc:442] T a6b4ebfad6324a63a56a77c559eed6cf P ee44c104bcf142e99c7bf4b4b46b6b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:33:33.177233 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.173274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.0.98.3:32945)
0927 16:33:33.173389 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.173393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.173614 (+ 221us) server_negotiation.cc:407] Connection header received
0927 16:33:33.173748 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.173750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.173802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.173880 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.174142 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.174711 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.175916 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.176120 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.176210 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.176471 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.176862 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.176945 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.177040 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.177085 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:33:33.188179 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.184711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58825 (local address 127.0.98.2:45561)
0927 16:33:33.184856 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.184861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.185111 (+ 250us) server_negotiation.cc:407] Connection header received
0927 16:33:33.185267 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.185269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.185320 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.185404 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.185668 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.186220 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.187081 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.187309 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.187412 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.187748 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.187817 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.187892 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.187984 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.188034 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:33:33.188198 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.184636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.3:58825)
0927 16:33:33.185020 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.185032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.185157 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.185415 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.185418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.185426 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:33.185549 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.185559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.186328 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.186334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.186959 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.186967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.187443 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.187449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.187537 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.187576 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.187907 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.187914 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.187974 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.188051 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":109,"threads_started":1}
I20260927 16:33:33.196467 2276 raft_consensus.cc:1261] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Refusing update from remote peer 8201c2a474334f9c916e0d86a6ae45e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.196800 2416 raft_consensus.cc:1261] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Refusing update from remote peer 8201c2a474334f9c916e0d86a6ae45e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.196944 2635 consensus_queue.cc:1035] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.197331 2632 consensus_queue.cc:1035] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.203536 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333997826847703040 to 7333997825599631360. Current Snapshot: MvccSnapshot[applied={T|T < 7333997826847703040}]
I20260927 16:33:33.220916 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.216769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55579 (local address 127.0.98.3:32945)
0927 16:33:33.216906 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.216911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.217105 (+ 194us) server_negotiation.cc:407] Connection header received
0927 16:33:33.217305 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.217311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.217369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.217447 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.217699 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.218453 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.219675 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.219955 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.220097 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.220405 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.220488 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.220570 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.220726 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.220782 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:33:33.220919 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.216701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32945 (local address 127.0.98.2:55579)
0927 16:33:33.217013 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.217029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.217180 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.217460 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.217463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.217471 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:33.217584 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.217590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.218597 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.218600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.219523 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.219535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.220102 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.220105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.220197 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.220250 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.220612 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.220619 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.220689 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.220771 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":122,"threads_started":1}
I20260927 16:33:33.250991 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:33.251195 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:33.254148 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Sending DeleteTablet for 3 replicas of tablet e688c250f3bb440e94c16e62b81fa3e9
I20260927 16:33:33.255030 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49552
I20260927 16:33:33.255594 2674 tablet_replica.cc:330] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: stopping tablet replica
I20260927 16:33:33.255913 2674 raft_consensus.cc:2229] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:33.256351 2674 raft_consensus.cc:2258] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:33.256558 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58918
I20260927 16:33:33.257175 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:33.257591 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:33.257886 2674 ts_tablet_manager.cc:1905] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:33.260924 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet e688c250f3bb440e94c16e62b81fa3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260927 16:33:33.261297 2674 ts_tablet_manager.cc:1918] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:33.261399 2674 log.cc:1196] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/e688c250f3bb440e94c16e62b81fa3e9
I20260927 16:33:33.261729 2674 ts_tablet_manager.cc:1939] T e688c250f3bb440e94c16e62b81fa3e9 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting consensus metadata
I20260927 16:33:33.263089 2106 catalog_manager.cc:4841] TS 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945): tablet e688c250f3bb440e94c16e62b81fa3e9 (table test-table [id=e5e34962b9504a5e90a3b9853a235e51]) successfully deleted
I20260927 16:33:33.263509 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 (DEFAULT_TABLE table=test-table [id=28294f7ed66d4a23be437e7b48fadbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:33.263792 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87525a5a150b4f9cbbb1cf02109ed246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:33.264142 2676 tablet_replica.cc:330] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: stopping tablet replica
I20260927 16:33:33.264434 2676 raft_consensus.cc:2229] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:33.264602 2676 raft_consensus.cc:2258] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:33.264892 2675 tablet_replica.cc:330] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: stopping tablet replica
I20260927 16:33:33.265180 2675 raft_consensus.cc:2229] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:33.265403 2675 raft_consensus.cc:2258] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:33.266244 2627 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Bootstrap starting.
I20260927 16:33:33.266589 2676 ts_tablet_manager.cc:1905] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:33.267179 2627 tablet_bootstrap.cc:654] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:33.268674 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 (DEFAULT_TABLE table=test-table [id=28294f7ed66d4a23be437e7b48fadbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:33.268951 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87525a5a150b4f9cbbb1cf02109ed246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:33.269451 2676 ts_tablet_manager.cc:1918] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:33.269534 2676 log.cc:1196] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/e688c250f3bb440e94c16e62b81fa3e9
I20260927 16:33:33.269840 2676 ts_tablet_manager.cc:1939] T e688c250f3bb440e94c16e62b81fa3e9 P b6416122765d43708c081993feca70af: Deleting consensus metadata
I20260927 16:33:33.271145 2105 catalog_manager.cc:4841] TS b6416122765d43708c081993feca70af (127.0.98.2:45561): tablet e688c250f3bb440e94c16e62b81fa3e9 (table test-table [id=e5e34962b9504a5e90a3b9853a235e51]) successfully deleted
I20260927 16:33:33.271725 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 (DEFAULT_TABLE table=test-table [id=28294f7ed66d4a23be437e7b48fadbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:33.271977 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87525a5a150b4f9cbbb1cf02109ed246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:33.272382 2625 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Bootstrap starting.
I20260927 16:33:33.273336 2625 tablet_bootstrap.cc:654] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:33.274147 2626 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Bootstrap starting.
I20260927 16:33:33.275116 2626 tablet_bootstrap.cc:654] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:33.275595 2675 ts_tablet_manager.cc:1905] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:33.277434 2626 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: No bootstrap required, opened a new log
I20260927 16:33:33.277516 2626 ts_tablet_manager.cc:1397] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:33:33.277941 2626 raft_consensus.cc:348] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.278052 2626 raft_consensus.cc:374] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:33.278083 2626 raft_consensus.cc:729] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6416122765d43708c081993feca70af, State: Initialized, Role: FOLLOWER
I20260927 16:33:33.278235 2626 consensus_queue.cc:260] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.278514 2626 ts_tablet_manager.cc:1428] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:33:33.278640 2675 ts_tablet_manager.cc:1918] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:33.278740 2675 log.cc:1196] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/e688c250f3bb440e94c16e62b81fa3e9
I20260927 16:33:33.279040 2675 ts_tablet_manager.cc:1939] T e688c250f3bb440e94c16e62b81fa3e9 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting consensus metadata
I20260927 16:33:33.280263 2106 catalog_manager.cc:4841] TS ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593): tablet e688c250f3bb440e94c16e62b81fa3e9 (table test-table [id=e5e34962b9504a5e90a3b9853a235e51]) successfully deleted
I20260927 16:33:33.280864 2627 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: No bootstrap required, opened a new log
I20260927 16:33:33.280944 2627 ts_tablet_manager.cc:1397] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260927 16:33:33.281229 2625 tablet_bootstrap.cc:492] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: No bootstrap required, opened a new log
I20260927 16:33:33.281303 2625 ts_tablet_manager.cc:1397] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 16:33:33.281347 2627 raft_consensus.cc:348] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.281445 2627 raft_consensus.cc:374] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:33.281474 2627 raft_consensus.cc:729] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Initialized, Role: FOLLOWER
I20260927 16:33:33.281594 2627 consensus_queue.cc:260] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.281822 2627 ts_tablet_manager.cc:1428] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:33:33.281747 2625 raft_consensus.cc:348] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.281873 2625 raft_consensus.cc:374] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:33.281911 2625 raft_consensus.cc:729] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Initialized, Role: FOLLOWER
I20260927 16:33:33.282086 2625 consensus_queue.cc:260] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.282411 2625 ts_tablet_manager.cc:1428] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:33.411499 2661 raft_consensus.cc:482] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:33.411703 2661 raft_consensus.cc:504] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.412210 2661 leader_election.cc:288] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:33.412753 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87525a5a150b4f9cbbb1cf02109ed246" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" is_pre_election: true
I20260927 16:33:33.412753 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87525a5a150b4f9cbbb1cf02109ed246" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6" is_pre_election: true
I20260927 16:33:33.412909 2276 raft_consensus.cc:2454] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 0.
I20260927 16:33:33.412918 2556 raft_consensus.cc:2454] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 0.
I20260927 16:33:33.413250 2350 leader_election.cc:302] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, b6416122765d43708c081993feca70af; no voters:
I20260927 16:33:33.413463 2661 raft_consensus.cc:2781] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:33.413594 2661 raft_consensus.cc:482] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:33.413664 2661 raft_consensus.cc:3037] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.414808 2661 raft_consensus.cc:504] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.415141 2661 leader_election.cc:288] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 election: Requested vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:33.415666 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87525a5a150b4f9cbbb1cf02109ed246" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6"
I20260927 16:33:33.415699 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87525a5a150b4f9cbbb1cf02109ed246" candidate_uuid: "b6416122765d43708c081993feca70af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
I20260927 16:33:33.415812 2556 raft_consensus.cc:3037] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.415812 2276 raft_consensus.cc:3037] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:33.416923 2276 raft_consensus.cc:2454] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 1.
I20260927 16:33:33.416923 2556 raft_consensus.cc:2454] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6416122765d43708c081993feca70af in term 1.
I20260927 16:33:33.417264 2350 leader_election.cc:302] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, b6416122765d43708c081993feca70af; no voters:
I20260927 16:33:33.417423 2661 raft_consensus.cc:2781] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:33.417536 2661 raft_consensus.cc:686] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 1 LEADER]: Becoming Leader. State: Replica: b6416122765d43708c081993feca70af, State: Running, Role: LEADER
I20260927 16:33:33.417702 2661 consensus_queue.cc:237] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:33.419452 2117 catalog_manager.cc:5495] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af reported cstate change: term changed from 0 to 1, leader changed from <none> to b6416122765d43708c081993feca70af (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b6416122765d43708c081993feca70af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20260927 16:33:33.440611 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.437527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.98.2:45561)
0927 16:33:33.437635 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.437641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.437701 (+ 60us) server_negotiation.cc:407] Connection header received
0927 16:33:33.437859 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.437862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.437908 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.437991 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.438263 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.438779 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.439597 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.439835 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.439937 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.440184 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.440262 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.440339 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.440425 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.440470 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:33:33.449156 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.446199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.2:58685)
0927 16:33:33.446320 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:33.446335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:33.446412 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:33.446631 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:33.446634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:33.446640 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:33.446722 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.446726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.447471 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.447477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.448076 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:33.448083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.448511 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.448514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:33.448591 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.448615 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.448875 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.448882 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:33:33.448946 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:33:33.449045 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":19}
I20260927 16:33:33.449164 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:33.446284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58685 (local address 127.0.98.2:45561)
0927 16:33:33.446368 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:33.446373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:33.446385 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:33:33.446507 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:33.446510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:33.446559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:33.446616 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:33.446853 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.447368 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.448189 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:33.448409 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:33.448515 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:33.448727 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:33.448798 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:33.448873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:33.448952 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 16:33:33.449006 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260927 16:33:33.454959 2276 raft_consensus.cc:1261] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Refusing update from remote peer b6416122765d43708c081993feca70af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.455057 2556 raft_consensus.cc:1261] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Refusing update from remote peer b6416122765d43708c081993feca70af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:33.455344 2661 consensus_queue.cc:1035] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.455488 2633 consensus_queue.cc:1035] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:33.460364 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333997827908579328 to 7333997827760562176. Current Snapshot: MvccSnapshot[applied={T|T < 7333997827908579328}]
W20260927 16:33:33.469759 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:33:35.022228 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:35.022367 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:35.024602 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Sending DeleteTablet for 3 replicas of tablet 87525a5a150b4f9cbbb1cf02109ed246
I20260927 16:33:35.025162 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260927 16:33:35.025213 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49552
I20260927 16:33:35.025207 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 87525a5a150b4f9cbbb1cf02109ed246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58918
I20260927 16:33:35.025534 2706 tablet_replica.cc:330] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: stopping tablet replica
I20260927 16:33:35.025523 2707 tablet_replica.cc:330] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: stopping tablet replica
I20260927 16:33:35.025668 2707 raft_consensus.cc:2229] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:35.025668 2706 raft_consensus.cc:2229] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:35.025779 2707 raft_consensus.cc:2258] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.025906 2706 raft_consensus.cc:2258] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.026453 2707 ts_tablet_manager.cc:1905] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.026530 2706 ts_tablet_manager.cc:1905] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.029186 2706 ts_tablet_manager.cc:1918] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:33:35.029186 2707 ts_tablet_manager.cc:1918] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:33:35.029270 2707 log.cc:1196] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/87525a5a150b4f9cbbb1cf02109ed246
I20260927 16:33:35.029270 2706 log.cc:1196] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/87525a5a150b4f9cbbb1cf02109ed246
I20260927 16:33:35.029558 2707 ts_tablet_manager.cc:1939] T 87525a5a150b4f9cbbb1cf02109ed246 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting consensus metadata
I20260927 16:33:35.029543 2706 ts_tablet_manager.cc:1939] T 87525a5a150b4f9cbbb1cf02109ed246 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting consensus metadata
I20260927 16:33:35.030262 2106 catalog_manager.cc:4841] TS 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945): tablet 87525a5a150b4f9cbbb1cf02109ed246 (table test-table [id=28294f7ed66d4a23be437e7b48fadbd7]) successfully deleted
I20260927 16:33:35.030440 2106 catalog_manager.cc:4841] TS ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593): tablet 87525a5a150b4f9cbbb1cf02109ed246 (table test-table [id=28294f7ed66d4a23be437e7b48fadbd7]) successfully deleted
I20260927 16:33:35.032655 2705 tablet_replica.cc:330] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: stopping tablet replica
I20260927 16:33:35.032809 2705 raft_consensus.cc:2229] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:35.033147 2705 raft_consensus.cc:2258] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.033751 2705 ts_tablet_manager.cc:1905] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.036309 2705 ts_tablet_manager.cc:1918] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:33:35.036374 2705 log.cc:1196] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/87525a5a150b4f9cbbb1cf02109ed246
I20260927 16:33:35.036623 2705 ts_tablet_manager.cc:1939] T 87525a5a150b4f9cbbb1cf02109ed246 P b6416122765d43708c081993feca70af: Deleting consensus metadata
I20260927 16:33:35.037359 2105 catalog_manager.cc:4841] TS b6416122765d43708c081993feca70af (127.0.98.2:45561): tablet 87525a5a150b4f9cbbb1cf02109ed246 (table test-table [id=28294f7ed66d4a23be437e7b48fadbd7]) successfully deleted
Sep 27 16:33:35 dist-test-slave-t27c krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:33:35 dist-test-slave-t27c krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:35.063875 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.048212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.62:34605)
0927 16:33:35.048484 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.048493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.048571 (+ 78us) server_negotiation.cc:407] Connection header received
0927 16:33:35.048752 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.048754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.048797 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.048889 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:35.049596 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.050148 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.051032 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.051268 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.051377 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.060565 (+ 9188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:35.060569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:35.060571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:35.060600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:35.062062 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:35.062687 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:35.062690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:35.062692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:35.062766 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:35.063050 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:35.063052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:35.063053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:35.063278 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:35.063382 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.063595 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.063698 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260927 16:33:35.066371 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:35.066831 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:35.071735 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 6e794de0037b42a1a7456bfddaff6452 (DEFAULT_TABLE table=test-table [id=c17113b6de374717930a623d71ae26be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.071928 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 6e794de0037b42a1a7456bfddaff6452 (DEFAULT_TABLE table=test-table [id=c17113b6de374717930a623d71ae26be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.071964 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 6e794de0037b42a1a7456bfddaff6452 (DEFAULT_TABLE table=test-table [id=c17113b6de374717930a623d71ae26be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.072062 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e794de0037b42a1a7456bfddaff6452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.072198 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e794de0037b42a1a7456bfddaff6452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.072252 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e794de0037b42a1a7456bfddaff6452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.074800 2716 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Bootstrap starting.
I20260927 16:33:35.074869 2718 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Bootstrap starting.
I20260927 16:33:35.075562 2716 tablet_bootstrap.cc:654] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.075575 2718 tablet_bootstrap.cc:654] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.076176 2717 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Bootstrap starting.
I20260927 16:33:35.076709 2716 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: No bootstrap required, opened a new log
I20260927 16:33:35.076850 2716 ts_tablet_manager.cc:1397] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:35.077041 2717 tablet_bootstrap.cc:654] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.077137 2718 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: No bootstrap required, opened a new log
I20260927 16:33:35.077229 2718 ts_tablet_manager.cc:1397] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:35.077523 2716 raft_consensus.cc:348] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.077673 2716 raft_consensus.cc:374] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.077715 2716 raft_consensus.cc:729] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.077659 2718 raft_consensus.cc:348] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.077771 2718 raft_consensus.cc:374] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.077816 2718 raft_consensus.cc:729] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.077895 2716 consensus_queue.cc:260] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.077955 2718 consensus_queue.cc:260] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.078326 2716 ts_tablet_manager.cc:1428] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:35.078329 2718 ts_tablet_manager.cc:1428] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:35.078655 2717 tablet_bootstrap.cc:492] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: No bootstrap required, opened a new log
I20260927 16:33:35.078753 2717 ts_tablet_manager.cc:1397] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:35.079107 2717 raft_consensus.cc:348] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.079236 2717 raft_consensus.cc:374] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.079305 2717 raft_consensus.cc:729] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6416122765d43708c081993feca70af, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.079460 2717 consensus_queue.cc:260] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.079761 2717 ts_tablet_manager.cc:1428] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:35.167390 2722 raft_consensus.cc:482] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:35.167569 2722 raft_consensus.cc:504] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.168666 2722 leader_election.cc:288] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6416122765d43708c081993feca70af (127.0.98.2:45561), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:35.172737 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.169172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49637 (local address 127.0.98.3:32945)
0927 16:33:35.169434 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.169451 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.169468 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:35.169595 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.169598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.169674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.169760 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:35.170048 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.170561 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.171749 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.172135 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.172257 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.172336 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.172511 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.172605 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260927 16:33:35.172840 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.168955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53449 (local address 127.0.98.2:45561)
0927 16:33:35.169324 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.169333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.169349 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:35.169494 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.169498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.169568 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.169649 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:35.169976 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.170593 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.171710 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.172127 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.172256 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.172321 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.172615 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.172693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
I20260927 16:33:35.172762 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.168818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.1:53449)
0927 16:33:35.169242 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:35.169262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:35.169373 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:35.169684 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:35.169687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:35.169712 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:35.169850 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.169857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.170728 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.170731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.171588 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.171597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.172370 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.172373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.172490 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.172519 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:33:35.172584 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:33:35.172637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":17,"thread_start_us":126,"threads_started":1}
I20260927 16:33:35.172760 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.168915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32945 (local address 127.0.98.1:49637)
0927 16:33:35.169363 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:35.169379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:35.169496 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:35.169793 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:35.169795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:35.169808 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:35.169899 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.169904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.170659 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.170665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.171598 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.171607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.172266 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.172269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.172360 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.172411 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 16:33:35.172483 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:33:35.172559 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":78,"thread_start_us":133,"threads_started":1}
I20260927 16:33:35.173372 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e794de0037b42a1a7456bfddaff6452" candidate_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af" is_pre_election: true
I20260927 16:33:35.173545 2416 raft_consensus.cc:2454] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee44c104bcf142e99c7bf4b4b46b6b9f in term 0.
I20260927 16:33:35.173607 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e794de0037b42a1a7456bfddaff6452" candidate_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6" is_pre_election: true
I20260927 16:33:35.173730 2556 raft_consensus.cc:2454] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee44c104bcf142e99c7bf4b4b46b6b9f in term 0.
I20260927 16:33:35.173925 2209 leader_election.cc:302] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:35.174239 2722 raft_consensus.cc:2781] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:35.174299 2722 raft_consensus.cc:482] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:35.174338 2722 raft_consensus.cc:3037] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.175374 2722 raft_consensus.cc:504] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.175738 2722 leader_election.cc:288] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [CANDIDATE]: Term 1 election: Requested vote from peers b6416122765d43708c081993feca70af (127.0.98.2:45561), 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945)
I20260927 16:33:35.176047 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e794de0037b42a1a7456bfddaff6452" candidate_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af"
I20260927 16:33:35.176160 2416 raft_consensus.cc:3037] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.176126 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e794de0037b42a1a7456bfddaff6452" candidate_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8201c2a474334f9c916e0d86a6ae45e6"
I20260927 16:33:35.176262 2556 raft_consensus.cc:3037] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.177078 2416 raft_consensus.cc:2454] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee44c104bcf142e99c7bf4b4b46b6b9f in term 1.
I20260927 16:33:35.177191 2556 raft_consensus.cc:2454] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee44c104bcf142e99c7bf4b4b46b6b9f in term 1.
I20260927 16:33:35.177326 2209 leader_election.cc:302] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:35.177488 2722 raft_consensus.cc:2781] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:35.177690 2722 raft_consensus.cc:686] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 LEADER]: Becoming Leader. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Running, Role: LEADER
I20260927 16:33:35.178071 2722 consensus_queue.cc:237] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.180365 2119 catalog_manager.cc:5495] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20260927 16:33:35.206588 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.202945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.1:41593)
0927 16:33:35.203208 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.203214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.203225 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:33:35.203295 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.203297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.203350 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.203458 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:35.203649 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.204181 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.205009 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.205314 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.205394 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.205655 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.206127 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.206291 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.206328 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.206401 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":115,"threads_started":1}
I20260927 16:33:35.216218 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.213397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48773 (local address 127.0.98.2:45561)
0927 16:33:35.213522 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.213526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.213539 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:35.213631 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.213633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.213675 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.213747 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:35.213964 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.214531 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.215303 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.215557 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.215635 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.215870 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.215930 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.215982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.216065 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.216108 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:35.216236 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.213148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.1:48773)
0927 16:33:35.213445 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:35.213458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:35.213558 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:35.213772 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:35.213774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:35.213780 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:35.213875 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.213879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.214621 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.214623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.215204 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.215211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.215652 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.215654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.215725 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.215766 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.215998 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.216002 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:33:35.216057 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 16:33:35.216116 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260927 16:33:35.222136 2416 raft_consensus.cc:1261] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Refusing update from remote peer ee44c104bcf142e99c7bf4b4b46b6b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:35.222146 2556 raft_consensus.cc:1261] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Refusing update from remote peer ee44c104bcf142e99c7bf4b4b46b6b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:35.222661 2722 consensus_queue.cc:1035] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:35.222844 2728 consensus_queue.cc:1035] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:35.242995 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.239301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36957 (local address 127.0.98.1:41593)
0927 16:33:35.239565 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.239569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.239633 (+ 64us) server_negotiation.cc:407] Connection header received
0927 16:33:35.239757 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.239760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.239818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.239896 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:35.240214 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.240738 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.241686 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.241970 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.242052 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.242404 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.242605 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.242696 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.242791 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.242849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260927 16:33:35.243227 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.239278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.98.2:36957)
0927 16:33:35.239544 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:35.239557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:35.239646 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:35.240002 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:35.240004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:35.240011 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:33:35.240123 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.240127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.240823 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.240826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.241591 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:35.241599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.242065 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.242068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:35.242225 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.242265 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.242699 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.242706 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:33:35.242799 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:33:35.242943 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
I20260927 16:33:35.266395 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:35.266577 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:35.268720 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Sending DeleteTablet for 3 replicas of tablet 6e794de0037b42a1a7456bfddaff6452
I20260927 16:33:35.269382 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e794de0037b42a1a7456bfddaff6452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260927 16:33:35.269531 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e794de0037b42a1a7456bfddaff6452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58918
I20260927 16:33:35.269590 2705 tablet_replica.cc:330] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: stopping tablet replica
I20260927 16:33:35.269724 2705 raft_consensus.cc:2229] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:35.269744 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e794de0037b42a1a7456bfddaff6452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49552
I20260927 16:33:35.269878 2705 raft_consensus.cc:2258] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.269922 2707 tablet_replica.cc:330] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: stopping tablet replica
I20260927 16:33:35.270026 2707 raft_consensus.cc:2229] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:35.270399 2707 raft_consensus.cc:2258] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.270622 2706 tablet_replica.cc:330] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: stopping tablet replica
I20260927 16:33:35.270748 2706 raft_consensus.cc:2229] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:35.270866 2706 raft_consensus.cc:2258] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:35.271248 2705 ts_tablet_manager.cc:1905] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.271556 2707 ts_tablet_manager.cc:1905] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.272308 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:35.272778 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:35.274062 2706 ts_tablet_manager.cc:1905] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:35.277405 2706 ts_tablet_manager.cc:1918] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:35.277501 2706 log.cc:1196] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/6e794de0037b42a1a7456bfddaff6452
I20260927 16:33:35.277756 2706 ts_tablet_manager.cc:1939] T 6e794de0037b42a1a7456bfddaff6452 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting consensus metadata
I20260927 16:33:35.280923 2256 tablet_service.cc:1467] Processing CreateTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 (DEFAULT_TABLE table=test-table [id=59bba1f267ad4c2ebd5593fa51398230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.281208 2396 tablet_service.cc:1467] Processing CreateTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 (DEFAULT_TABLE table=test-table [id=59bba1f267ad4c2ebd5593fa51398230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.281629 2536 tablet_service.cc:1467] Processing CreateTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 (DEFAULT_TABLE table=test-table [id=59bba1f267ad4c2ebd5593fa51398230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:35.281877 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b941eb889b415cbebaf4d7b0c50997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.284057 2718 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Bootstrap starting.
I20260927 16:33:35.284914 2718 tablet_bootstrap.cc:654] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.285986 2718 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: No bootstrap required, opened a new log
I20260927 16:33:35.286065 2718 ts_tablet_manager.cc:1397] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:35.286471 2718 raft_consensus.cc:348] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.286587 2718 raft_consensus.cc:374] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.286633 2718 raft_consensus.cc:729] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.286787 2718 consensus_queue.cc:260] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.287045 2718 ts_tablet_manager.cc:1428] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:35.291116 2106 catalog_manager.cc:4841] TS 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945): tablet 6e794de0037b42a1a7456bfddaff6452 (table test-table [id=c17113b6de374717930a623d71ae26be]) successfully deleted
I20260927 16:33:35.292532 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b941eb889b415cbebaf4d7b0c50997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.292542 2707 ts_tablet_manager.cc:1918] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:35.292675 2707 log.cc:1196] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/6e794de0037b42a1a7456bfddaff6452
I20260927 16:33:35.293001 2707 ts_tablet_manager.cc:1939] T 6e794de0037b42a1a7456bfddaff6452 P ee44c104bcf142e99c7bf4b4b46b6b9f: Deleting consensus metadata
I20260927 16:33:35.295043 2716 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: Bootstrap starting.
I20260927 16:33:35.296034 2716 tablet_bootstrap.cc:654] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.297300 2716 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: No bootstrap required, opened a new log
I20260927 16:33:35.297392 2716 ts_tablet_manager.cc:1397] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:35.297817 2716 raft_consensus.cc:348] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.297948 2716 raft_consensus.cc:374] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.297988 2716 raft_consensus.cc:729] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c104bcf142e99c7bf4b4b46b6b9f, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.298121 2716 consensus_queue.cc:260] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.298424 2716 ts_tablet_manager.cc:1428] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:35.299240 2106 catalog_manager.cc:4841] TS ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593): tablet 6e794de0037b42a1a7456bfddaff6452 (table test-table [id=c17113b6de374717930a623d71ae26be]) successfully deleted
I20260927 16:33:35.302790 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b941eb889b415cbebaf4d7b0c50997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:35.302825 2705 ts_tablet_manager.cc:1918] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:33:35.302887 2705 log.cc:1196] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/6e794de0037b42a1a7456bfddaff6452
I20260927 16:33:35.303182 2705 ts_tablet_manager.cc:1939] T 6e794de0037b42a1a7456bfddaff6452 P b6416122765d43708c081993feca70af: Deleting consensus metadata
I20260927 16:33:35.303912 2105 catalog_manager.cc:4841] TS b6416122765d43708c081993feca70af (127.0.98.2:45561): tablet 6e794de0037b42a1a7456bfddaff6452 (table test-table [id=c17113b6de374717930a623d71ae26be]) successfully deleted
I20260927 16:33:35.305380 2717 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Bootstrap starting.
I20260927 16:33:35.306846 2717 tablet_bootstrap.cc:654] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:35.307924 2717 tablet_bootstrap.cc:492] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: No bootstrap required, opened a new log
I20260927 16:33:35.308038 2717 ts_tablet_manager.cc:1397] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:35.308418 2717 raft_consensus.cc:348] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.308540 2717 raft_consensus.cc:374] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:35.308610 2717 raft_consensus.cc:729] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6416122765d43708c081993feca70af, State: Initialized, Role: FOLLOWER
I20260927 16:33:35.308782 2717 consensus_queue.cc:260] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.309073 2717 ts_tablet_manager.cc:1428] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:33:35.322670 2723 raft_consensus.cc:482] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:35.322814 2723 raft_consensus.cc:504] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.323271 2723 leader_election.cc:288] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), b6416122765d43708c081993feca70af (127.0.98.2:45561)
I20260927 16:33:35.323645 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b941eb889b415cbebaf4d7b0c50997" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" is_pre_election: true
I20260927 16:33:35.323731 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b941eb889b415cbebaf4d7b0c50997" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af" is_pre_election: true
I20260927 16:33:35.323804 2276 raft_consensus.cc:2454] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 0.
I20260927 16:33:35.323855 2416 raft_consensus.cc:2454] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 0.
I20260927 16:33:35.324155 2489 leader_election.cc:302] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:35.324385 2723 raft_consensus.cc:2781] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:35.324433 2723 raft_consensus.cc:482] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:35.324499 2723 raft_consensus.cc:3037] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.397375 2723 raft_consensus.cc:504] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.397893 2723 leader_election.cc:288] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 election: Requested vote from peers ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593), b6416122765d43708c081993feca70af (127.0.98.2:45561)
I20260927 16:33:35.398406 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b941eb889b415cbebaf4d7b0c50997" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f"
I20260927 16:33:35.398406 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b941eb889b415cbebaf4d7b0c50997" candidate_uuid: "8201c2a474334f9c916e0d86a6ae45e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6416122765d43708c081993feca70af"
I20260927 16:33:35.398550 2416 raft_consensus.cc:3037] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.398550 2276 raft_consensus.cc:3037] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:35.399780 2276 raft_consensus.cc:2454] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 1.
I20260927 16:33:35.399780 2416 raft_consensus.cc:2454] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8201c2a474334f9c916e0d86a6ae45e6 in term 1.
I20260927 16:33:35.400202 2490 leader_election.cc:302] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8201c2a474334f9c916e0d86a6ae45e6, b6416122765d43708c081993feca70af, ee44c104bcf142e99c7bf4b4b46b6b9f; no voters:
I20260927 16:33:35.400516 2723 raft_consensus.cc:2781] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:35.400779 2723 raft_consensus.cc:686] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8201c2a474334f9c916e0d86a6ae45e6, State: Running, Role: LEADER
I20260927 16:33:35.400957 2723 consensus_queue.cc:237] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } }
I20260927 16:33:35.402953 2117 catalog_manager.cc:5495] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8201c2a474334f9c916e0d86a6ae45e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8201c2a474334f9c916e0d86a6ae45e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32945 } health_report { overall_health: HEALTHY } } }
I20260927 16:33:35.447177 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:35.443955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.98.3:32945)
0927 16:33:35.444067 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:35.444072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:35.444090 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:33:35.444189 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:35.444191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:35.444232 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:35.444293 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:35.444528 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.445180 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.446096 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:35.446390 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:35.446469 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:35.446712 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:35.446776 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:35.446829 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:35.446949 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 16:33:35.447017 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:33:35.453739 2276 raft_consensus.cc:1261] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Refusing update from remote peer 8201c2a474334f9c916e0d86a6ae45e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:35.453799 2416 raft_consensus.cc:1261] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Refusing update from remote peer 8201c2a474334f9c916e0d86a6ae45e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:35.454236 2723 consensus_queue.cc:1035] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6416122765d43708c081993feca70af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:35.454396 2745 consensus_queue.cc:1035] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c104bcf142e99c7bf4b4b46b6b9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:33:35.467051 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:33:37.017148 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:37.017315 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:37.019865 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b393ef5ba7049fb8e1ecd172b30431a: Sending DeleteTablet for 3 replicas of tablet f7b941eb889b415cbebaf4d7b0c50997
I20260927 16:33:37.020341 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58918
I20260927 16:33:37.020416 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49552
I20260927 16:33:37.020722 2774 tablet_replica.cc:330] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f: stopping tablet replica
I20260927 16:33:37.020711 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b941eb889b415cbebaf4d7b0c50997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60274
I20260927 16:33:37.020875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2191
I20260927 16:33:37.020854 2774 raft_consensus.cc:2229] T f7b941eb889b415cbebaf4d7b0c50997 P ee44c104bcf142e99c7bf4b4b46b6b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:37.021437 2776 tablet_replica.cc:330] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: stopping tablet replica
I20260927 16:33:37.021586 2776 raft_consensus.cc:2229] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:37.021698 2776 raft_consensus.cc:2258] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:37.022440 2776 ts_tablet_manager.cc:1905] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:37.025801 2776 ts_tablet_manager.cc:1918] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:33:37.025902 2776 log.cc:1196] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/f7b941eb889b415cbebaf4d7b0c50997
I20260927 16:33:37.026113 2775 tablet_replica.cc:330] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: stopping tablet replica
I20260927 16:33:37.026201 2776 ts_tablet_manager.cc:1939] T f7b941eb889b415cbebaf4d7b0c50997 P b6416122765d43708c081993feca70af: Deleting consensus metadata
I20260927 16:33:37.026283 2775 raft_consensus.cc:2229] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:37.026492 2775 raft_consensus.cc:2258] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:37.027081 2775 ts_tablet_manager.cc:1905] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:37.027091 2105 catalog_manager.cc:4841] TS b6416122765d43708c081993feca70af (127.0.98.2:45561): tablet f7b941eb889b415cbebaf4d7b0c50997 (table test-table [id=59bba1f267ad4c2ebd5593fa51398230]) successfully deleted
I20260927 16:33:37.029038 2775 ts_tablet_manager.cc:1918] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:33:37.029131 2775 log.cc:1196] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/f7b941eb889b415cbebaf4d7b0c50997
I20260927 16:33:37.029460 2775 ts_tablet_manager.cc:1939] T f7b941eb889b415cbebaf4d7b0c50997 P 8201c2a474334f9c916e0d86a6ae45e6: Deleting consensus metadata
W20260927 16:33:37.029874 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41593 (error 108)
I20260927 16:33:37.030077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2331
I20260927 16:33:37.032467 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:37.032040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.0.1:57368)
0927 16:33:37.032292 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:37.032373 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
W20260927 16:33:37.032739 2106 catalog_manager.cc:4584] TS ee44c104bcf142e99c7bf4b4b46b6b9f (127.0.98.1:41593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7b941eb889b415cbebaf4d7b0c50997: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41593: connect: Connection refused (error 111)
I20260927 16:33:37.035210 2106 catalog_manager.cc:4841] TS 8201c2a474334f9c916e0d86a6ae45e6 (127.0.98.3:32945): tablet f7b941eb889b415cbebaf4d7b0c50997 (table test-table [id=59bba1f267ad4c2ebd5593fa51398230]) successfully deleted
I20260927 16:33:37.037650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2471
I20260927 16:33:37.044895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2087
2026-09-27T16:33:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6605 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:37 dist-test-slave-t27c krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:37 dist-test-slave-t27c krb5kdc[2781](info): set up 2 sockets
Sep 27 16:33:37 dist-test-slave-t27c krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260927 16:33:38.086118 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:38.209444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:38.319207 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:38.319499 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:38.319541 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:38.323527 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:38.323585 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:38.323603 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:38.323621 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:38.323635 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:38.328989 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:38.329978 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:38.330991 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:38.339713 2794 server_base.cc:1034] running on GCE node
W20260927 16:33:38.339794 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:38.339723 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:38.339732 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:38.340492 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:38.341547 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:38.342726 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790526818342720 us; error 37 us; skew 500 ppm
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:38.345285 2794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:38.346108 2794 webserver.cc:460] Webserver started at http://127.0.98.62:35763/ using document root <none> and password file <none>
I20260927 16:33:38.346673 2794 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:38.346784 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:38.347005 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:38.348978 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "bc7007bb1f7f43daae13f61d61e61041"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "420e3d7856b61f500a85c0f3b0ae4652"
server_key_iv: "cb2b63c1d677eb7f59856bd223be66ba"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.349531 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc7007bb1f7f43daae13f61d61e61041"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "420e3d7856b61f500a85c0f3b0ae4652"
server_key_iv: "cb2b63c1d677eb7f59856bd223be66ba"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.353350 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:33:38.355931 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.357223 2794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:38.357388 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "bc7007bb1f7f43daae13f61d61e61041"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "420e3d7856b61f500a85c0f3b0ae4652"
server_key_iv: "cb2b63c1d677eb7f59856bd223be66ba"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.357507 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:38.372502 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:38.375360 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:38.375554 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:38.382200 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41113
I20260927 16:33:38.382273 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41113 every 8 connection(s)
I20260927 16:33:38.383157 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:38.385243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2794
I20260927 16:33:38.385370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:38.385638 392 external_mini_cluster.cc:1428] Setting key 682417527c9c357a20afead99a846c78
I20260927 16:33:38.385979 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:38.393273 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041: Bootstrap starting.
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:38.396052 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:38.397006 2862 log.cc:824] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:38.399302 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041: No bootstrap required, opened a new log
I20260927 16:33:38.400709 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:38.387044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.98.62:41113)
0927 16:33:38.387461 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:38.387480 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:38.387521 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:33:38.388292 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:38.388315 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:38.388730 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:38.389097 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:38.389749 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.391233 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.392092 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.392799 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.392965 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:38.395694 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:38.395700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:38.395717 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:38.395750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:38.397682 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.398117 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.398123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.398129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.398313 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.398568 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.398571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.398572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.399068 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:38.399260 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:38.399476 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 16:33:38.399774 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260927 16:33:38.403553 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } }
I20260927 16:33:38.403760 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:38.403803 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7007bb1f7f43daae13f61d61e61041, State: Initialized, Role: FOLLOWER
I20260927 16:33:38.404161 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } }
I20260927 16:33:38.404269 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:38.404320 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:38.404389 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:38.405207 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } }
I20260927 16:33:38.405490 2862 leader_election.cc:302] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7007bb1f7f43daae13f61d61e61041; no voters:
I20260927 16:33:38.405735 2862 leader_election.cc:288] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:38.405896 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:38.406134 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [term 1 LEADER]: Becoming Leader. State: Replica: bc7007bb1f7f43daae13f61d61e61041, State: Running, Role: LEADER
I20260927 16:33:38.406477 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } }
I20260927 16:33:38.406935 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:38.408070 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7007bb1f7f43daae13f61d61e61041. Latest consensus state: current_term: 1 leader_uuid: "bc7007bb1f7f43daae13f61d61e61041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } } }
I20260927 16:33:38.408223 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:38.408593 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:38.409250 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7007bb1f7f43daae13f61d61e61041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7007bb1f7f43daae13f61d61e61041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41113 } } }
I20260927 16:33:38.409435 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:38.411705 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:38.417322 2876 catalog_manager.cc:1347] Generated new cluster ID: 7aba38a895c84b69bbc7972fc3349ef8
I20260927 16:33:38.417385 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:38.430485 2876 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:38.431746 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:38.438747 2876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7007bb1f7f43daae13f61d61e61041: Generated new TSK 0
I20260927 16:33:38.439612 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:38.468659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:38.579926 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:38.580188 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:38.580229 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:38.584403 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:38.584468 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:38.584556 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:38.589927 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:38.591113 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:38.592175 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:38.601675 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:38.601815 2888 server_base.cc:1034] running on GCE node
W20260927 16:33:38.601682 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:38.601720 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:38.602440 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:38.603085 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:38.604239 2888 hybrid_clock.cc:648] HybridClock initialized: now 1790526818604206 us; error 46 us; skew 500 ppm
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:38.606851 2888 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:38.607759 2888 webserver.cc:460] Webserver started at http://127.0.98.1:38251/ using document root <none> and password file <none>
I20260927 16:33:38.608311 2888 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:38.608407 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:38.608603 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:38.610615 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "be983e8b07fb4834a2dcf3901fd41267"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "073151add5c5c6b51a44156bc409f00e"
server_key_iv: "c8e4eed7979eaeda114818a6106ca109"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.611116 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be983e8b07fb4834a2dcf3901fd41267"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "073151add5c5c6b51a44156bc409f00e"
server_key_iv: "c8e4eed7979eaeda114818a6106ca109"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.614625 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:38.616887 2903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.618155 2888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:38.618299 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "be983e8b07fb4834a2dcf3901fd41267"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "073151add5c5c6b51a44156bc409f00e"
server_key_iv: "c8e4eed7979eaeda114818a6106ca109"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.618381 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:38.636318 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:38.639514 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:38.639684 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:38.640308 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:38.641112 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:38.641160 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.641199 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:38.641220 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.650415 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35265
I20260927 16:33:38.650444 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35265 every 8 connection(s)
I20260927 16:33:38.651311 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:38.654808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 2888
I20260927 16:33:38.654946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:38.655246 392 external_mini_cluster.cc:1428] Setting key 2d1b7b87ffefec9f306e3f41ee23da24
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:38.665679 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:38.653034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40619 (local address 127.0.98.62:41113)
0927 16:33:38.653222 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:38.653226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:38.654001 (+ 775us) server_negotiation.cc:407] Connection header received
0927 16:33:38.655844 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:38.655846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:38.655891 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:38.655993 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:38.657438 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.657955 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.659207 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.659453 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.659566 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:38.662036 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:38.662040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:38.662042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:38.662070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:38.663618 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.664149 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.664155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.664157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.664243 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.664581 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.664586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.664588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.664775 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:38.664881 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:38.665370 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 16:33:38.665496 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 16:33:38.666548 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:38.653423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41113 (local address 127.0.98.1:40619)
0927 16:33:38.653855 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:38.653897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:38.655614 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:38.656139 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:38.656148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:38.656606 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:38.657285 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:38.657308 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.658120 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.658125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:38.659084 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:38.659093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.659611 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.659615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:38.659776 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:38.660377 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:38.660393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:38.661884 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:38.663760 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:38.663766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:38.663781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:38.664032 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:38.664347 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:38.664350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:38.664352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:38.664487 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:38.664877 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:38.664883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:38.665131 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 16:33:38.665319 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 16:33:38.665644 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:33:38.667809 3017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41113
I20260927 16:33:38.668066 3017 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:38.668659 3017 heartbeater.cc:511] Master 127.0.98.62:41113 requested a full tablet report, sending...
I20260927 16:33:38.670258 2826 ts_manager.cc:194] Registered new tserver with Master: be983e8b07fb4834a2dcf3901fd41267 (127.0.98.1:35265)
I20260927 16:33:38.671408 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:38.682828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:38.792631 3022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:38.792948 3022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:38.792994 3022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:38.797151 3022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:38.797223 3022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:38.797283 3022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:38.802687 3022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:38.803658 3022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:38.804629 3022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:38.814074 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:38.814098 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:38.814074 3030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:38.814304 3022 server_base.cc:1034] running on GCE node
I20260927 16:33:38.814723 3022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:38.815351 3022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:38.816519 3022 hybrid_clock.cc:648] HybridClock initialized: now 1790526818816495 us; error 36 us; skew 500 ppm
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:38.819291 3022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:38.820159 3022 webserver.cc:460] Webserver started at http://127.0.98.2:46503/ using document root <none> and password file <none>
I20260927 16:33:38.820700 3022 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:38.820798 3022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:38.821023 3022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:38.823101 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "52f0a9ac2713466abae3cb56468fdf6c"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "5a403df34d06c06c529e299c54d9a413"
server_key_iv: "7d128d4968d737ebcc5a00108980fd65"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.823658 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52f0a9ac2713466abae3cb56468fdf6c"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "5a403df34d06c06c529e299c54d9a413"
server_key_iv: "7d128d4968d737ebcc5a00108980fd65"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.827199 3022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:33:38.829625 3037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.830868 3022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:38.830999 3022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "52f0a9ac2713466abae3cb56468fdf6c"
format_stamp: "Formatted at 2026-09-27 16:33:38 on dist-test-slave-t27c"
server_key: "5a403df34d06c06c529e299c54d9a413"
server_key_iv: "7d128d4968d737ebcc5a00108980fd65"
server_key_version: "encryptionkey@0"
I20260927 16:33:38.831112 3022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:38.850955 3022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:38.854246 3022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:38.854452 3022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:38.855074 3022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:38.855888 3022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:38.855957 3022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.856030 3022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:38.856074 3022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:38.865100 3022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38415
I20260927 16:33:38.865123 3150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38415 every 8 connection(s)
I20260927 16:33:38.866030 3022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:38.869174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3022
I20260927 16:33:38.869266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:38.869554 392 external_mini_cluster.cc:1428] Setting key 706a17d9672cea4678b403b67ef38e39
Sep 27 16:33:38 dist-test-slave-t27c krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:33:38.881529 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:38.867830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50525 (local address 127.0.98.62:41113)
0927 16:33:38.867970 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:38.867979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:38.868842 (+ 863us) server_negotiation.cc:407] Connection header received
0927 16:33:38.870607 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:38.870610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:38.870652 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:38.870721 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:38.872014 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.872858 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.874532 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.874943 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.875070 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:38.877914 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:38.877920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:38.877923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:38.877956 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:38.879608 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.880114 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.880117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.880118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.880194 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:38.880488 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:38.880490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:38.880491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:38.880660 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:38.880774 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:38.881295 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 16:33:38.881398 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:33:38.882556 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:38.868255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41113 (local address 127.0.98.2:50525)
0927 16:33:38.868683 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:38.868734 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:38.870387 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:38.870913 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:38.870924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:38.871352 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:38.871851 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:38.871865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.873005 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.873011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:38.874401 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:38.874412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:38.875106 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:38.875109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:38.875344 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:38.875969 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:38.875986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:38.877738 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:38.879735 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:38.879742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:38.879768 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:38.880019 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:38.880282 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:38.880284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:38.880285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:38.880412 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:38.880785 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:38.880794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:38.881057 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 16:33:38.881229 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 16:33:38.881500 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260927 16:33:38.883879 3151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41113
I20260927 16:33:38.884177 3151 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:38.884783 3151 heartbeater.cc:511] Master 127.0.98.62:41113 requested a full tablet report, sending...
I20260927 16:33:38.885931 2826 ts_manager.cc:194] Registered new tserver with Master: 52f0a9ac2713466abae3cb56468fdf6c (127.0.98.2:38415)
I20260927 16:33:38.886592 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:38.897727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:39.007493 3156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:39.007773 3156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:39.007854 3156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:39.012125 3156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:39.012207 3156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:39.012293 3156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:39.017477 3156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:39.018626 3156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:39.019613 3156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:39.029439 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:39.029484 3156 server_base.cc:1034] running on GCE node
W20260927 16:33:39.029434 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:39.029429 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:39.030131 3156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:39.030867 3156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:39.032076 3156 hybrid_clock.cc:648] HybridClock initialized: now 1790526819032016 us; error 28 us; skew 500 ppm
Sep 27 16:33:39 dist-test-slave-t27c krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:39.039418 3156 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:39.040460 3156 webserver.cc:460] Webserver started at http://127.0.98.3:43283/ using document root <none> and password file <none>
I20260927 16:33:39.041138 3156 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:39.041213 3156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:39.041437 3156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:39.043659 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "f7c39cbf98914d518af81653f09b7c0d"
format_stamp: "Formatted at 2026-09-27 16:33:39 on dist-test-slave-t27c"
server_key: "d3fcd67bbadd6452a1097ccf975d2fea"
server_key_iv: "db3c99415b3da82f049468b90e0e4122"
server_key_version: "encryptionkey@0"
I20260927 16:33:39.044286 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f7c39cbf98914d518af81653f09b7c0d"
format_stamp: "Formatted at 2026-09-27 16:33:39 on dist-test-slave-t27c"
server_key: "d3fcd67bbadd6452a1097ccf975d2fea"
server_key_iv: "db3c99415b3da82f049468b90e0e4122"
server_key_version: "encryptionkey@0"
I20260927 16:33:39.048148 3156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:33:39.050693 3171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:39.051966 3156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:39.052145 3156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "f7c39cbf98914d518af81653f09b7c0d"
format_stamp: "Formatted at 2026-09-27 16:33:39 on dist-test-slave-t27c"
server_key: "d3fcd67bbadd6452a1097ccf975d2fea"
server_key_iv: "db3c99415b3da82f049468b90e0e4122"
server_key_version: "encryptionkey@0"
I20260927 16:33:39.052268 3156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:39.084071 3156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:39.087921 3156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:39.088148 3156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:39.088819 3156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:39.089706 3156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:39.089780 3156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:39.089848 3156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:39.089898 3156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:39.098455 3156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44589
I20260927 16:33:39.098480 3284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44589 every 8 connection(s)
I20260927 16:33:39.099668 3156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:39.103987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3156
I20260927 16:33:39.104125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:39.104374 392 external_mini_cluster.cc:1428] Setting key f9d6fc5190f74e788b2356e5bd7705c0
Sep 27 16:33:39 dist-test-slave-t27c krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:39.114046 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:39.101545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52681 (local address 127.0.98.62:41113)
0927 16:33:39.101704 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:39.101708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:39.102544 (+ 836us) server_negotiation.cc:407] Connection header received
0927 16:33:39.103889 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:39.103893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:39.103952 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:39.104067 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:39.105462 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:39.106090 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:39.107364 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:39.107695 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:39.107778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:39.110317 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:39.110321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:39.110323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:39.110356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:39.111864 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:39.112405 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:39.112410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:39.112412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:39.112511 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:39.112896 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:39.112900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:39.112901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:39.113095 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:39.113204 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:39.113732 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 16:33:39.113863 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 16:33:39.114979 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:39.101942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41113 (local address 127.0.98.3:52681)
0927 16:33:39.102398 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:39.102440 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:39.103685 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:39.104288 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:39.104294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:39.104729 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:39.105326 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:39.105341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:39.106271 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:39.106276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:39.107255 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:39.107264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:39.107829 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:39.107832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:39.107974 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:39.108564 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:39.108580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:39.110072 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:39.111991 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:39.111997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:39.112014 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:39.112295 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:39.112627 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:39.112632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:39.112635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:39.112791 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:39.113201 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:39.113210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:39.113484 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 16:33:39.113693 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 16:33:39.114048 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260927 16:33:39.116246 3285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41113
I20260927 16:33:39.116642 3285 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:39.117275 3285 heartbeater.cc:511] Master 127.0.98.62:41113 requested a full tablet report, sending...
I20260927 16:33:39.118461 2826 ts_manager.cc:194] Registered new tserver with Master: f7c39cbf98914d518af81653f09b7c0d (127.0.98.3:44589)
I20260927 16:33:39.119004 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52681
I20260927 16:33:39.130095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:33:39.142899 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:39.141476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.0.98.62:41113)
0927 16:33:39.141618 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:39.141622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:39.141800 (+ 178us) server_negotiation.cc:407] Connection header received
0927 16:33:39.141914 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:39.141917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:39.141959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:39.142039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:39.142748 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55164: BlockingRecv error: recv got EOF from 127.0.0.1:55164 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:33:39.144238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2888
I20260927 16:33:39.149641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3022
I20260927 16:33:39.155107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3156
I20260927 16:33:39.160274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 2794
2026-09-27T16:33:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2113 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:39 dist-test-slave-t27c krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:39 dist-test-slave-t27c krb5kdc[3297](info): set up 2 sockets
Sep 27 16:33:39 dist-test-slave-t27c krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260927 16:33:40.206161 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:40.329243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:40.437991 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:40.438396 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:40.438486 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:40.442679 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:40.442756 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:40.442802 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:40.442837 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:40.442868 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:40.448184 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:40.449249 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:40.450304 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:40.458904 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:40.458881 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:40.458878 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:40.459141 3310 server_base.cc:1034] running on GCE node
I20260927 16:33:40.459661 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:40.460737 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:40.461895 3310 hybrid_clock.cc:648] HybridClock initialized: now 1790526820461862 us; error 48 us; skew 500 ppm
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:40.470799 3310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:40.471829 3310 webserver.cc:460] Webserver started at http://127.0.98.62:34887/ using document root <none> and password file <none>
I20260927 16:33:40.472429 3310 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:40.472523 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:40.472775 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:40.475101 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "0b148e5c7ca14a9ca00fe2f2b179352f"
format_stamp: "Formatted at 2026-09-27 16:33:40 on dist-test-slave-t27c"
server_key: "63391861b8bd11276efdc23d2a35219a"
server_key_iv: "5596d47f6f7f9f1627d45c40d9b9e766"
server_key_version: "encryptionkey@0"
I20260927 16:33:40.475693 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b148e5c7ca14a9ca00fe2f2b179352f"
format_stamp: "Formatted at 2026-09-27 16:33:40 on dist-test-slave-t27c"
server_key: "63391861b8bd11276efdc23d2a35219a"
server_key_iv: "5596d47f6f7f9f1627d45c40d9b9e766"
server_key_version: "encryptionkey@0"
I20260927 16:33:40.479528 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:33:40.481889 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:40.483233 3310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:40.483382 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "0b148e5c7ca14a9ca00fe2f2b179352f"
format_stamp: "Formatted at 2026-09-27 16:33:40 on dist-test-slave-t27c"
server_key: "63391861b8bd11276efdc23d2a35219a"
server_key_iv: "5596d47f6f7f9f1627d45c40d9b9e766"
server_key_version: "encryptionkey@0"
I20260927 16:33:40.483493 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:40.508189 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:40.511111 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:40.511327 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:40.518676 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36399
I20260927 16:33:40.518980 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36399 every 8 connection(s)
I20260927 16:33:40.519721 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:40.522527 3378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:40.525453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3310
I20260927 16:33:40.525574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:40.525856 392 external_mini_cluster.cc:1428] Setting key 4913324b92973b0d44d7e817001f0bb0
I20260927 16:33:40.528365 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: Bootstrap starting.
I20260927 16:33:40.530871 3378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:40.531756 3378 log.cc:824] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:40.533921 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: No bootstrap required, opened a new log
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:40.537937 3378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } }
I20260927 16:33:40.538182 3378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:40.538261 3378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b148e5c7ca14a9ca00fe2f2b179352f, State: Initialized, Role: FOLLOWER
I20260927 16:33:40.538725 3378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } }
I20260927 16:33:40.538944 3378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:40.539016 3378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:40.539104 3378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:40.540181 3378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } }
I20260927 16:33:40.540513 3378 leader_election.cc:302] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b148e5c7ca14a9ca00fe2f2b179352f; no voters:
I20260927 16:33:40.540812 3378 leader_election.cc:288] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:40.540918 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:40.540944 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:40.527143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.98.62:36399)
0927 16:33:40.527549 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:40.527564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:40.527600 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:33:40.528208 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:40.528224 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:40.528590 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:40.528870 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:40.529771 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:40.530820 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:40.531979 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:40.532436 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:40.532550 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:40.535226 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:40.535232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:40.535247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:40.535276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:40.537776 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:40.538255 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:40.538261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:40.538268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:40.538404 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:40.538718 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:40.538721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:40.538723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:40.539277 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:40.539472 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:40.539723 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 16:33:40.540014 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260927 16:33:40.541177 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [term 1 LEADER]: Becoming Leader. State: Replica: 0b148e5c7ca14a9ca00fe2f2b179352f, State: Running, Role: LEADER
I20260927 16:33:40.541494 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } }
I20260927 16:33:40.541973 3378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:40.542711 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b148e5c7ca14a9ca00fe2f2b179352f. Latest consensus state: current_term: 1 leader_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } } }
I20260927 16:33:40.542841 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:40.543123 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b148e5c7ca14a9ca00fe2f2b179352f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36399 } } }
I20260927 16:33:40.543241 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:40.544358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 16:33:40.546806 3398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:40.546887 3398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:40.546947 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:40.547763 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:40.553972 3392 catalog_manager.cc:1347] Generated new cluster ID: 511a6335ca344218b587ab01f1aed22c
I20260927 16:33:40.554050 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:40.567022 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:40.562154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.0.98.62:36399)
0927 16:33:40.562307 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:40.562311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:40.562565 (+ 254us) server_negotiation.cc:407] Connection header received
0927 16:33:40.562741 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:40.562744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:40.562795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:40.562882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:40.563392 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:40.563964 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:40.564809 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:40.565048 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:40.565164 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:40.566020 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:40.566024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:40.566027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:40.566553 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:40.566581 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:40.566597 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:40.566697 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:40.566791 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 16:33:40.566853 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260927 16:33:40.574100 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:40.574997 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:40.583949 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b148e5c7ca14a9ca00fe2f2b179352f: Generated new TSK 0
I20260927 16:33:40.584695 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:33:40.598796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3310
2026-09-27T16:33:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1436 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3411](info): set up 2 sockets
Sep 27 16:33:40 dist-test-slave-t27c krb5kdc[3411](info): commencing operation
krb5kdc: starting...
W20260927 16:33:41.634091 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:41 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:41.753175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:41.863023 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:41.863274 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:41.863313 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:41.867229 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:41.867285 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:41.867303 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:41.867321 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:41.867334 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:41.872445 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:41.873481 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:41.874560 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:41.883036 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:41.883008 3430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:41.883159 3424 server_base.cc:1034] running on GCE node
W20260927 16:33:41.883332 3432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:41.883867 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:41.884923 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:41.886080 3424 hybrid_clock.cc:648] HybridClock initialized: now 1790526821886065 us; error 27 us; skew 500 ppm
Sep 27 16:33:41 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:41.888839 3424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:41.889709 3424 webserver.cc:460] Webserver started at http://127.0.98.62:42343/ using document root <none> and password file <none>
I20260927 16:33:41.890282 3424 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:41.890343 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:41.890503 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:41.892391 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "867b217bc226475fbb95393345b096de"
format_stamp: "Formatted at 2026-09-27 16:33:41 on dist-test-slave-t27c"
server_key: "4d2a41deb3e848814cec328f1424affe"
server_key_iv: "9059c7abc1624fe7ed40e90b4cf823c0"
server_key_version: "encryptionkey@0"
I20260927 16:33:41.892870 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "867b217bc226475fbb95393345b096de"
format_stamp: "Formatted at 2026-09-27 16:33:41 on dist-test-slave-t27c"
server_key: "4d2a41deb3e848814cec328f1424affe"
server_key_iv: "9059c7abc1624fe7ed40e90b4cf823c0"
server_key_version: "encryptionkey@0"
I20260927 16:33:41.896313 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:33:41.898499 3439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:41.899646 3424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:41.899773 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "867b217bc226475fbb95393345b096de"
format_stamp: "Formatted at 2026-09-27 16:33:41 on dist-test-slave-t27c"
server_key: "4d2a41deb3e848814cec328f1424affe"
server_key_iv: "9059c7abc1624fe7ed40e90b4cf823c0"
server_key_version: "encryptionkey@0"
I20260927 16:33:41.899874 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:41.921742 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:41.925216 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:41.925424 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:41.932214 3424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36565
I20260927 16:33:41.932498 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36565 every 8 connection(s)
I20260927 16:33:41.933161 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:41.935904 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:41.939160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3424
I20260927 16:33:41.939297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:41.939564 392 external_mini_cluster.cc:1428] Setting key 67006bf499c262ab66c618a53e0e85d4
I20260927 16:33:41.941826 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de: Bootstrap starting.
I20260927 16:33:41.944533 3492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:41.945369 3492 log.cc:824] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:41.947567 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de: No bootstrap required, opened a new log
Sep 27 16:33:41 dist-test-slave-t27c krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:41.951686 3492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260927 16:33:41.951967 3492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:41.952018 3492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867b217bc226475fbb95393345b096de, State: Initialized, Role: FOLLOWER
I20260927 16:33:41.952503 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260927 16:33:41.952661 3492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:41.952744 3492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:41.952859 3492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:41.953886 3492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260927 16:33:41.954309 3492 leader_election.cc:302] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 867b217bc226475fbb95393345b096de; no voters:
I20260927 16:33:41.954633 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:41.940896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.0.98.62:36565)
0927 16:33:41.941305 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:41.941324 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:41.941366 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:33:41.942189 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:41.942213 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:41.942612 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:41.942987 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:41.943600 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:41.945118 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:41.946267 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:41.946909 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:41.947086 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:41.949399 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:41.949405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:41.949422 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:41.949453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:41.951458 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:41.951864 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:41.951870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:41.951876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:41.952033 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:41.952333 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:41.952337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:41.952339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:41.952839 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:41.953059 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:41.953358 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:33:41.953657 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260927 16:33:41.954672 3492 leader_election.cc:288] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:41.954749 3497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:41.954939 3497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [term 1 LEADER]: Becoming Leader. State: Replica: 867b217bc226475fbb95393345b096de, State: Running, Role: LEADER
I20260927 16:33:41.955327 3497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260927 16:33:41.956029 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:41.957049 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 867b217bc226475fbb95393345b096de. Latest consensus state: current_term: 1 leader_uuid: "867b217bc226475fbb95393345b096de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260927 16:33:41.957191 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:41.957407 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "867b217bc226475fbb95393345b096de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867b217bc226475fbb95393345b096de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260927 16:33:41.957482 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:41.958220 3506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:41.960980 3506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:41.966099 3506 catalog_manager.cc:1347] Generated new cluster ID: b22effbacb764533a1a7c86275e05fed
I20260927 16:33:41.966184 3506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:41.979600 3506 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:41.980618 3506 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:41.986625 3506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 867b217bc226475fbb95393345b096de: Generated new TSK 0
I20260927 16:33:41.987267 3506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:42.016667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:42.128860 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:42.129122 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:42.129164 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:42.133237 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:42.133291 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:42.133378 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:42.138722 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:42.139860 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:42.140920 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:42.150872 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:42.150904 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:42.150854 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:42.151211 3518 server_base.cc:1034] running on GCE node
I20260927 16:33:42.151604 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:42.152339 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:42.153533 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790526822153516 us; error 38 us; skew 500 ppm
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:42.157296 3518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:42.158512 3518 webserver.cc:460] Webserver started at http://127.0.98.1:40653/ using document root <none> and password file <none>
I20260927 16:33:42.159276 3518 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:42.159379 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:42.159633 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:42.162068 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "afc70d69083b4829a5a6566a4f0f75c2"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "df10db8959875ef1f568f54f0ff1e094"
server_key_iv: "ba2308a83d8b2eb235aa3671a958dead"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.162792 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "afc70d69083b4829a5a6566a4f0f75c2"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "df10db8959875ef1f568f54f0ff1e094"
server_key_iv: "ba2308a83d8b2eb235aa3671a958dead"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.167292 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:33:42.169970 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.171319 3518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:42.171460 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "afc70d69083b4829a5a6566a4f0f75c2"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "df10db8959875ef1f568f54f0ff1e094"
server_key_iv: "ba2308a83d8b2eb235aa3671a958dead"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.171584 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:42.189401 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:42.192566 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:42.192783 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:42.193434 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:42.194382 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:42.194453 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.194527 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:42.194583 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.204666 3646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44101 every 8 connection(s)
I20260927 16:33:42.204667 3518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44101
I20260927 16:33:42.205827 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:42.212924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3518
I20260927 16:33:42.213022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:42.213321 392 external_mini_cluster.cc:1428] Setting key f53af1a373ad74dbdf42df6525dbcabe
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:42.221035 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.207763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34857 (local address 127.0.98.62:36565)
0927 16:33:42.207927 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:42.207932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:42.208824 (+ 892us) server_negotiation.cc:407] Connection header received
0927 16:33:42.210158 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:42.210160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:42.210259 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:42.210334 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:42.211432 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.211968 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.213106 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.213456 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.213582 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.216792 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:42.216798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:42.216805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:42.216835 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:42.218707 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.219216 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.219222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.219224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.219322 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.219651 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.219657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.219660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.220002 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:42.220118 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:42.220634 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 16:33:42.220788 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 16:33:42.221701 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.208194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:34857)
0927 16:33:42.208667 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:42.208711 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:42.209964 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:42.210477 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:42.210484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:42.210835 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:42.211304 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.211317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.212072 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.212074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.212950 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.212958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.213621 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.213625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.213798 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.214399 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:42.214420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:42.216628 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:42.218819 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.218824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.218836 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.219121 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.219427 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.219429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.219431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.219570 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.220111 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:42.220116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:42.220351 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 16:33:42.220513 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 16:33:42.220778 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260927 16:33:42.223013 3647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260927 16:33:42.223248 3647 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:42.223734 3647 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
I20260927 16:33:42.225297 3456 ts_manager.cc:194] Registered new tserver with Master: afc70d69083b4829a5a6566a4f0f75c2 (127.0.98.1:44101)
I20260927 16:33:42.226568 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 16:33:42.442289 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.227s user 0.002s sys 0.001s
I20260927 16:33:42.442793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:42.554469 3652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:42.554759 3652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:42.554804 3652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:42.559219 3652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:42.559285 3652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:42.559348 3652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:42.564770 3652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:42.565840 3652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:42.566972 3652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:42.576265 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:42.576249 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:42.576413 3652 server_base.cc:1034] running on GCE node
W20260927 16:33:42.576249 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:42.576922 3652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:42.577550 3652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:42.578723 3652 hybrid_clock.cc:648] HybridClock initialized: now 1790526822578713 us; error 28 us; skew 500 ppm
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:42.581310 3652 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:42.582132 3652 webserver.cc:460] Webserver started at http://127.0.98.2:40235/ using document root <none> and password file <none>
I20260927 16:33:42.582718 3652 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:42.582809 3652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:42.583041 3652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:42.585000 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "319ded46d7964e4b988054f677c26fd4"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "d95236eefa561495691e077df41ab5a6"
server_key_iv: "be872540c6f29119490cb980bf48d9fc"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.585554 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "319ded46d7964e4b988054f677c26fd4"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "d95236eefa561495691e077df41ab5a6"
server_key_iv: "be872540c6f29119490cb980bf48d9fc"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.589219 3652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:33:42.591593 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.592777 3652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:42.592911 3652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "319ded46d7964e4b988054f677c26fd4"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "d95236eefa561495691e077df41ab5a6"
server_key_iv: "be872540c6f29119490cb980bf48d9fc"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.593024 3652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:42.616101 3652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:42.618949 3652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:42.619210 3652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:42.619809 3652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:42.620625 3652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:42.620695 3652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.620766 3652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:42.620816 3652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.629462 3652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46081
I20260927 16:33:42.629487 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46081 every 8 connection(s)
I20260927 16:33:42.630385 3652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:42.639631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3652
I20260927 16:33:42.639761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:42.640071 392 external_mini_cluster.cc:1428] Setting key f3781cc4d07c3ebf43342d57de309f8c
I20260927 16:33:42.644546 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.632194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49147 (local address 127.0.98.62:36565)
0927 16:33:42.632340 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:42.632344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:42.633250 (+ 906us) server_negotiation.cc:407] Connection header received
0927 16:33:42.634691 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:42.634693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:42.634741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:42.634816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:42.636011 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.636544 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.637665 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.637876 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.637954 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.640740 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:42.640743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:42.640745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:42.640769 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:42.642417 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.642945 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.642952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.642957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.643050 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.643413 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.643417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.643419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.643614 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:42.643740 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:42.644246 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 16:33:42.644381 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:42.645406 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.632638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:49147)
0927 16:33:42.633093 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:42.633137 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:42.634486 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:42.635001 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:42.635008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:42.635375 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:42.635872 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.635885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.636662 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.636664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.637563 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.637571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.638013 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.638016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.638162 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.638835 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:42.638852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:42.640562 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:42.642551 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.642557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.642576 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.642835 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.643141 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.643143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.643145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.643310 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.643723 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:42.643742 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:42.644004 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 16:33:42.644206 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 16:33:42.644561 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260927 16:33:42.646904 3781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260927 16:33:42.647205 3781 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:42.647688 3781 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
I20260927 16:33:42.648789 3456 ts_manager.cc:194] Registered new tserver with Master: 319ded46d7964e4b988054f677c26fd4 (127.0.98.2:46081)
I20260927 16:33:42.649358 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49147
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:42.667177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:42.779143 3786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:42.779429 3786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:42.779538 3786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:42.783731 3786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:42.783798 3786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:42.783890 3786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:42.789191 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:42.790319 3786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:42.791366 3786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:42.801375 3786 server_base.cc:1034] running on GCE node
W20260927 16:33:42.801296 3794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:42.801306 3792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:42.801301 3791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:42.802008 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:42.802825 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:42.803982 3786 hybrid_clock.cc:648] HybridClock initialized: now 1790526822803955 us; error 40 us; skew 500 ppm
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:42.806746 3786 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:42.807606 3786 webserver.cc:460] Webserver started at http://127.0.98.3:42919/ using document root <none> and password file <none>
I20260927 16:33:42.808168 3786 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:42.808228 3786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:42.808459 3786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:42.810565 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "159e5ac7248e4027b086c63b94fd4be8"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "2343c54a9c0fd739d91cf963d9402bf6"
server_key_iv: "ce28b08642bcdfda95871f2f4c5c4f52"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.811120 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "159e5ac7248e4027b086c63b94fd4be8"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "2343c54a9c0fd739d91cf963d9402bf6"
server_key_iv: "ce28b08642bcdfda95871f2f4c5c4f52"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.814903 3786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:33:42.817301 3801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.818567 3786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:42.818703 3786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "159e5ac7248e4027b086c63b94fd4be8"
format_stamp: "Formatted at 2026-09-27 16:33:42 on dist-test-slave-t27c"
server_key: "2343c54a9c0fd739d91cf963d9402bf6"
server_key_iv: "ce28b08642bcdfda95871f2f4c5c4f52"
server_key_version: "encryptionkey@0"
I20260927 16:33:42.818814 3786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:42.839337 3786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:42.842386 3786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:42.842617 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:42.843297 3786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:42.844178 3786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:42.844254 3786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.844328 3786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:42.844383 3786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:42.853495 3786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35161
I20260927 16:33:42.853516 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35161 every 8 connection(s)
I20260927 16:33:42.854480 3786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:42.863672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3786
I20260927 16:33:42.863817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:42.864126 392 external_mini_cluster.cc:1428] Setting key 0969ef60b625fd13f336d349f36a01dc
I20260927 16:33:42.868301 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.856244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47787 (local address 127.0.98.62:36565)
0927 16:33:42.856383 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:42.856387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:42.857393 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 16:33:42.858943 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:42.858946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:42.858990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:42.859070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:42.860198 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.860719 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.861794 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.862005 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.862078 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.864760 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:42.864763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:42.864765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:42.864812 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:42.866507 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.866962 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.866965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.866966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.867037 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.867342 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.867345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.867347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.867538 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:42.867627 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:42.868074 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 16:33:42.868175 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:42.869175 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.856719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:47787)
0927 16:33:42.857237 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:42.857291 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:42.858740 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:42.859229 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:42.859236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:42.859584 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:42.860073 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.860086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.860820 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.860822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.861692 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:42.861699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.862131 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.862133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:42.862348 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.862874 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:42.862888 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:42.864560 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:42.866608 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.866612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.866624 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.866875 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.867111 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:42.867113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:42.867115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:42.867239 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:42.867631 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:42.867636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:42.867859 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 16:33:42.868021 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 16:33:42.868269 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260927 16:33:42.870396 3915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260927 16:33:42.870641 3915 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:42.871112 3915 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
I20260927 16:33:42.872097 3456 ts_manager.cc:194] Registered new tserver with Master: 159e5ac7248e4027b086c63b94fd4be8 (127.0.98.3:35161)
I20260927 16:33:42.872707 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47787
I20260927 16:33:42.878154 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526822, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:42.902390 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:42.894393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.98.62:36565)
0927 16:33:42.894556 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:42.894561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:42.894700 (+ 139us) server_negotiation.cc:407] Connection header received
0927 16:33:42.894809 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:42.894811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:42.894855 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:42.894939 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:42.895598 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.896177 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.896952 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:42.897191 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:42.897303 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:42.899673 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:42.899676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:42.899678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:42.899699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:42.900884 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.901246 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.901248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.901249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.901318 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:42.901543 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:42.901545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:42.901546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:42.901748 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:42.901849 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:42.902037 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 16:33:42.902127 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 16:33:42.902789 3456 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60498
I20260927 16:33:42.903743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3518
I20260927 16:33:42.909688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3652
I20260927 16:33:42.915164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3786
I20260927 16:33:42.920501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3424
2026-09-27T16:33:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2324 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3927](info): set up 2 sockets
Sep 27 16:33:42 dist-test-slave-t27c krb5kdc[3927](info): commencing operation
krb5kdc: starting...
W20260927 16:33:43.966143 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:44.085009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:44.206560 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:44.206838 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:44.206918 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:44.211266 3940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:44.211326 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:44.211347 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:44.211391 3940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:44.211426 3940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:44.217361 3940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:44.218499 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:44.219532 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:44.228101 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.228123 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.228173 3940 server_base.cc:1034] running on GCE node
W20260927 16:33:44.228418 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.228962 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:44.230043 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:44.231244 3940 hybrid_clock.cc:648] HybridClock initialized: now 1790526824231225 us; error 30 us; skew 500 ppm
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:44.233898 3940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:44.234805 3940 webserver.cc:460] Webserver started at http://127.0.98.62:46371/ using document root <none> and password file <none>
I20260927 16:33:44.235332 3940 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:44.235392 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:44.235558 3940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:44.237638 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "76bd8370905d4654b105f2d2c1d916c6"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "81c57ccc691706a4b2907d49ebf02a7f"
server_key_iv: "d194c72d7c93bbc0e487ff07f69f6a49"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.238227 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "76bd8370905d4654b105f2d2c1d916c6"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "81c57ccc691706a4b2907d49ebf02a7f"
server_key_iv: "d194c72d7c93bbc0e487ff07f69f6a49"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.241940 3940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:33:44.244370 3955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.245590 3940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:44.245718 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "76bd8370905d4654b105f2d2c1d916c6"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "81c57ccc691706a4b2907d49ebf02a7f"
server_key_iv: "d194c72d7c93bbc0e487ff07f69f6a49"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.245822 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:44.265879 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:44.268987 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:44.269194 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:44.276618 3940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37747
I20260927 16:33:44.276641 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37747 every 8 connection(s)
I20260927 16:33:44.277637 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:44.280393 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:44.281428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 3940
I20260927 16:33:44.281554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:44.281843 392 external_mini_cluster.cc:1428] Setting key abef56e6433d2c8e98ba5763c1da0055
I20260927 16:33:44.287070 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6: Bootstrap starting.
I20260927 16:33:44.289600 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:44.290400 4008 log.cc:824] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:44.292568 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6: No bootstrap required, opened a new log
I20260927 16:33:44.297152 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:44.283253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.98.62:37747)
0927 16:33:44.283688 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:44.283709 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:44.283752 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:33:44.284576 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:44.284598 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:44.285026 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:44.285390 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:44.286062 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.287491 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.288315 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.288979 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.289127 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:44.291805 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:44.291812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:44.291827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:44.291856 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:44.293768 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.294199 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.294205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.294211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.294351 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.294606 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.294610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.294611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.295191 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:44.295411 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:44.295598 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 16:33:44.295941 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":105,"threads_started":1}
I20260927 16:33:44.297510 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 16:33:44.297783 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:44.297869 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bd8370905d4654b105f2d2c1d916c6, State: Initialized, Role: FOLLOWER
I20260927 16:33:44.298365 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 16:33:44.298528 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:44.298627 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:44.298755 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:44.299851 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 16:33:44.300210 4008 leader_election.cc:302] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76bd8370905d4654b105f2d2c1d916c6; no voters:
I20260927 16:33:44.300546 4008 leader_election.cc:288] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:44.300627 4013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:44.300906 4013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [term 1 LEADER]: Becoming Leader. State: Replica: 76bd8370905d4654b105f2d2c1d916c6, State: Running, Role: LEADER
I20260927 16:33:44.301271 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 16:33:44.301913 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:44.302983 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76bd8370905d4654b105f2d2c1d916c6. Latest consensus state: current_term: 1 leader_uuid: "76bd8370905d4654b105f2d2c1d916c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260927 16:33:44.303108 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:44.303452 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76bd8370905d4654b105f2d2c1d916c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76bd8370905d4654b105f2d2c1d916c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260927 16:33:44.303546 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:44.303584 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:44.307094 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:44.312973 4022 catalog_manager.cc:1347] Generated new cluster ID: a5efc320b449487b9c1d1f89035da872
I20260927 16:33:44.313087 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:44.348888 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:44.350060 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:44.355243 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76bd8370905d4654b105f2d2c1d916c6: Generated new TSK 0
I20260927 16:33:44.356099 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:44.388896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:44.498788 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:44.499028 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:44.499068 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:44.503157 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:44.503247 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:44.503324 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:44.508579 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:44.509585 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:44.510633 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:44.520244 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.520233 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.520290 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.520434 4034 server_base.cc:1034] running on GCE node
I20260927 16:33:44.520965 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:44.521695 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:44.522878 4034 hybrid_clock.cc:648] HybridClock initialized: now 1790526824522843 us; error 48 us; skew 500 ppm
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:44.525583 4034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:44.526554 4034 webserver.cc:460] Webserver started at http://127.0.98.1:41355/ using document root <none> and password file <none>
I20260927 16:33:44.527118 4034 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:44.527207 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:44.527434 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:44.529656 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4bd7c26b3d84808b201d5456db6848d"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "dbcb0278a278211ee23c259c448ec84a"
server_key_iv: "405606e58efca2bad69f575e5a7eb481"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.530308 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4bd7c26b3d84808b201d5456db6848d"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "dbcb0278a278211ee23c259c448ec84a"
server_key_iv: "405606e58efca2bad69f575e5a7eb481"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.533885 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:44.536304 4049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.537451 4034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:33:44.537582 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "c4bd7c26b3d84808b201d5456db6848d"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "dbcb0278a278211ee23c259c448ec84a"
server_key_iv: "405606e58efca2bad69f575e5a7eb481"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.537689 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:44.556598 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:44.559664 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:44.559885 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:44.560485 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:44.561303 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:44.561376 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.561446 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:44.561499 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.570559 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37439
I20260927 16:33:44.570577 4162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37439 every 8 connection(s)
I20260927 16:33:44.571421 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:44.574714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4034
I20260927 16:33:44.574864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:44.575194 392 external_mini_cluster.cc:1428] Setting key f1e1285288520b34c8160fb66ea4e260
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:44.585500 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:44.573121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41331 (local address 127.0.98.62:37747)
0927 16:33:44.573253 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:44.573258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:44.573999 (+ 741us) server_negotiation.cc:407] Connection header received
0927 16:33:44.575824 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:44.575827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:44.575872 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:44.575975 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:44.577454 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.577979 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.579170 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.579411 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.579523 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:44.581846 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:44.581850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:44.581855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:44.581885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:44.583550 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.584044 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.584048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.584050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.584140 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.584465 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.584469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.584472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.584714 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:44.584817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:44.585243 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 16:33:44.585381 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:33:44.586516 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:44.573425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.1:41331)
0927 16:33:44.573852 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:44.573895 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:44.575584 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:44.576134 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:44.576141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:44.576623 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:44.577300 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:44.577323 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.578110 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.578120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:44.579048 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:44.579057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.579556 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.579560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:44.579724 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:44.580330 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:44.580345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:44.581683 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:44.583685 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:44.583689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:44.583699 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:44.583945 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:44.584229 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:44.584232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:44.584233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:44.584352 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:44.584838 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:44.584843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:44.585048 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 16:33:44.585193 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 16:33:44.585555 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:33:44.587749 4163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 16:33:44.587991 4163 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:44.588469 4163 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 16:33:44.589913 3972 ts_manager.cc:194] Registered new tserver with Master: c4bd7c26b3d84808b201d5456db6848d (127.0.98.1:37439)
I20260927 16:33:44.591107 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:44.601352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:44.710515 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:44.710826 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:44.710906 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:44.714946 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:44.715054 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:44.715150 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:44.720649 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:44.721792 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:44.722854 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:44.732259 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.732440 4168 server_base.cc:1034] running on GCE node
W20260927 16:33:44.732295 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.732295 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.733021 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:44.733700 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:44.734892 4168 hybrid_clock.cc:648] HybridClock initialized: now 1790526824734855 us; error 46 us; skew 500 ppm
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:44.737684 4168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:44.738597 4168 webserver.cc:460] Webserver started at http://127.0.98.2:38385/ using document root <none> and password file <none>
I20260927 16:33:44.739145 4168 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:44.739241 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:44.739471 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:44.741533 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "449955102ea946c5a705f16c0093c390"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "78bc943ae508535401385dcd731c90c4"
server_key_iv: "6f3df493f1aa93329ee3a1bcc51c8e0c"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.742105 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "449955102ea946c5a705f16c0093c390"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "78bc943ae508535401385dcd731c90c4"
server_key_iv: "6f3df493f1aa93329ee3a1bcc51c8e0c"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.745657 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:44.748109 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.749300 4168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:44.749420 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "449955102ea946c5a705f16c0093c390"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "78bc943ae508535401385dcd731c90c4"
server_key_iv: "6f3df493f1aa93329ee3a1bcc51c8e0c"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.749536 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:44.773471 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:44.776420 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:44.776625 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:44.777223 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:44.778085 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:44.778154 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.778247 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:44.778294 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.786715 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43977
I20260927 16:33:44.786736 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43977 every 8 connection(s)
I20260927 16:33:44.787645 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:44.787856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4168
I20260927 16:33:44.787974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:44.788311 392 external_mini_cluster.cc:1428] Setting key 5296be10cf22797e2b1277e75936baee
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:44.805397 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:44.789566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52293 (local address 127.0.98.62:37747)
0927 16:33:44.789717 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:44.789721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:44.790773 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 16:33:44.792503 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:44.792506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:44.792559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:44.792640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:44.793871 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.794486 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.795647 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.795973 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.796100 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:44.801365 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:44.801368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:44.801370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:44.801396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:44.803188 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.803744 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.803750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.803755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.803838 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:44.804264 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:44.804267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:44.804269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:44.804462 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:44.804574 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:44.805091 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 16:33:44.805223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:44.806318 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:44.790044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.2:52293)
0927 16:33:44.790596 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:44.790659 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:44.792290 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:44.792820 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:44.792827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:44.793173 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:44.793718 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:44.793747 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.794607 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.794611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:44.795519 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:44.795528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:44.796167 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:44.796174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:44.796419 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:44.797051 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:44.797079 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:44.801172 (+ 4093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:44.803327 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:44.803333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:44.803350 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:44.803634 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:44.803951 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:44.803957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:44.803961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:44.804158 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:44.804572 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:44.804578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:44.804851 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 16:33:44.805052 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 16:33:44.805397 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260927 16:33:44.807538 4297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 16:33:44.807808 4297 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:44.808319 4297 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 16:33:44.809350 3972 ts_manager.cc:194] Registered new tserver with Master: 449955102ea946c5a705f16c0093c390 (127.0.98.2:43977)
I20260927 16:33:44.809934 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:44.814680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:44.928695 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:44.928964 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:44.929008 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:44.933073 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:44.933127 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:44.933185 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:44.938274 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:44.939185 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:44.940171 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:44.949579 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.949579 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:44.949780 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:44.949851 4302 server_base.cc:1034] running on GCE node
I20260927 16:33:44.950284 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:44.950874 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:44.952083 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790526824952068 us; error 34 us; skew 500 ppm
Sep 27 16:33:44 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:44.954887 4302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:44.955785 4302 webserver.cc:460] Webserver started at http://127.0.98.3:46595/ using document root <none> and password file <none>
I20260927 16:33:44.956382 4302 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:44.956472 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:44.956703 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:44.958912 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "c227c3fdc6a64ec9aa46f711e1a8fb8c"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "e98812c8f5d7029aeddc20fa85d0da13"
server_key_iv: "b691df819450c7d219f4bd6f720e2ce0"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.959478 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c227c3fdc6a64ec9aa46f711e1a8fb8c"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "e98812c8f5d7029aeddc20fa85d0da13"
server_key_iv: "b691df819450c7d219f4bd6f720e2ce0"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.963217 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:33:44.965626 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.966887 4302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:44.967016 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "c227c3fdc6a64ec9aa46f711e1a8fb8c"
format_stamp: "Formatted at 2026-09-27 16:33:44 on dist-test-slave-t27c"
server_key: "e98812c8f5d7029aeddc20fa85d0da13"
server_key_iv: "b691df819450c7d219f4bd6f720e2ce0"
server_key_version: "encryptionkey@0"
I20260927 16:33:44.967120 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:44.984421 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:44.987426 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:44.987627 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:44.988194 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:44.989054 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:44.989161 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.989262 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:44.989315 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:44.998296 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36661
I20260927 16:33:44.998315 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36661 every 8 connection(s)
I20260927 16:33:44.999249 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:45.000564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4302
I20260927 16:33:45.000694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:45.001003 392 external_mini_cluster.cc:1428] Setting key c3a238e2dffd28b0c7f60ad0affaf039
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:45.012993 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:45.001372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38715 (local address 127.0.98.62:37747)
0927 16:33:45.001514 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:45.001518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:45.002460 (+ 942us) server_negotiation.cc:407] Connection header received
0927 16:33:45.003984 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:45.003986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:45.004029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:45.004090 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:45.005248 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.005762 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.006970 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.007178 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.007251 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:45.009745 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:45.009748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:45.009750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:45.009772 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:45.011196 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.011645 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.011648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.011649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.011721 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.012037 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.012041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.012043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.012248 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:45.012334 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:45.012757 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 16:33:45.012857 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:33:45.013876 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:45.001721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.3:38715)
0927 16:33:45.002309 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:45.002364 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:45.003791 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:45.004247 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:45.004254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:45.004632 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:45.005123 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:45.005137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.005891 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.005895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:45.006856 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:45.006865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.007297 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.007299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:45.007432 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:45.008080 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:45.008096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:45.009593 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:45.011295 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:45.011299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:45.011311 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:45.011555 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:45.011799 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:45.011801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:45.011802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:45.011929 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:45.012341 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:45.012345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:45.012554 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:33:45.012705 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 16:33:45.012953 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":146,"threads_started":1}
I20260927 16:33:45.015173 4431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 16:33:45.015415 4431 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:45.015903 4431 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 16:33:45.016862 3971 ts_manager.cc:194] Registered new tserver with Master: c227c3fdc6a64ec9aa46f711e1a8fb8c (127.0.98.3:36661)
I20260927 16:33:45.017455 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38715
I20260927 16:33:45.026661 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:45.052842 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:45.043399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.0.98.1:37439)
0927 16:33:45.043674 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:45.043688 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:45.043712 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:33:45.043783 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:45.043788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:45.043997 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:45.044123 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:45.044768 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.045535 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.046456 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.046854 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.046937 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:45.049340 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:45.049347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:45.049360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:45.049382 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:45.050958 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.051413 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.051418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.051419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.051497 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.051785 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.051795 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.051797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.052021 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:45.052261 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:45.052464 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 16:33:45.052646 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:45.063056 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:45.055822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.62:37747)
0927 16:33:45.055951 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:45.055955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:45.056019 (+ 64us) server_negotiation.cc:407] Connection header received
0927 16:33:45.056097 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:45.056098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:45.056138 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:45.056205 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:45.056721 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.057208 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.057912 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:45.058114 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:45.058299 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:45.060493 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:45.060496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:45.060498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:45.060519 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:45.061679 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.062038 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.062040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.062041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.062108 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:45.062386 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:45.062388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:45.062389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:45.062581 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:45.062679 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:45.062859 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 16:33:45.062942 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 16:33:45.063459 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51916
I20260927 16:33:45.064105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4034
I20260927 16:33:45.069871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4168
I20260927 16:33:45.075392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4302
I20260927 16:33:45.080924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 3940
2026-09-27T16:33:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2161 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[4443](info): set up 2 sockets
Sep 27 16:33:45 dist-test-slave-t27c krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260927 16:33:46.142247 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:46.262236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:46.371212 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:46.371519 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:46.371562 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:46.375528 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:46.375586 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:46.375610 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:46.375627 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:46.375643 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:46.380939 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:46.381930 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:46.382920 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:46.390985 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.390995 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.390995 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:46.391294 4456 server_base.cc:1034] running on GCE node
I20260927 16:33:46.391902 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:46.393003 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:46.394229 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790526826394199 us; error 48 us; skew 500 ppm
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:46.396939 4456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:46.397753 4456 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:46.397850 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:46.398092 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:46.400189 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "cbe08093326a6380bdf6aa7a499a6da1"
server_key_iv: "80455bcb8f24f58fc7ae365f4b32debb"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.400770 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "cbe08093326a6380bdf6aa7a499a6da1"
server_key_iv: "80455bcb8f24f58fc7ae365f4b32debb"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.404484 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:33:46.406914 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.408109 4456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:46.408250 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "cbe08093326a6380bdf6aa7a499a6da1"
server_key_iv: "80455bcb8f24f58fc7ae365f4b32debb"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.408360 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:46.427371 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:46.430235 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:46.430434 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:46.437354 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37241
I20260927 16:33:46.437646 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37241 every 8 connection(s)
I20260927 16:33:46.438100 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:46.440887 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:46.446303 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: Bootstrap starting.
I20260927 16:33:46.448184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4456
I20260927 16:33:46.448328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:46.448588 392 external_mini_cluster.cc:1428] Setting key e1caaab9184049aa97dc805063b0478b
I20260927 16:33:46.448791 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:46.449637 4523 log.cc:824] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:46.451522 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: No bootstrap required, opened a new log
I20260927 16:33:46.454886 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } }
I20260927 16:33:46.455133 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:46.455204 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4f9fd28da44f9b94c9e5fa0a8884af, State: Initialized, Role: FOLLOWER
I20260927 16:33:46.455657 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } }
I20260927 16:33:46.455780 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:46.455863 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:46.455960 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:46.457114 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } }
I20260927 16:33:46.457484 4523 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4f9fd28da44f9b94c9e5fa0a8884af; no voters:
I20260927 16:33:46.457830 4523 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:46.457923 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:46.458207 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [term 1 LEADER]: Becoming Leader. State: Replica: 1d4f9fd28da44f9b94c9e5fa0a8884af, State: Running, Role: LEADER
I20260927 16:33:46.458552 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } }
I20260927 16:33:46.459261 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:46.460069 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } } }
I20260927 16:33:46.460487 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:46.460386 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4f9fd28da44f9b94c9e5fa0a8884af. Latest consensus state: current_term: 1 leader_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4f9fd28da44f9b94c9e5fa0a8884af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37241 } } }
I20260927 16:33:46.460807 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:46.463279 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:46.449696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.98.62:37241)
0927 16:33:46.450155 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:46.450176 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:46.450237 (+ 61us) server_negotiation.cc:407] Connection header received
0927 16:33:46.450909 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:46.450926 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:46.451236 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:46.451568 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:46.452332 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.453271 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.454119 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.454640 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.454815 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:46.457757 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:46.457767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:46.457786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:46.457817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:46.459940 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.460549 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.460556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.460561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.460696 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.461037 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.461041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.461042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.461600 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:46.461813 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:46.462307 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 16:33:46.462587 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
W20260927 16:33:46.464395 4543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:46.464468 4543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:46.464531 4537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:46.465168 4537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:46.470971 4537 catalog_manager.cc:1347] Generated new cluster ID: cedb4b1f83ab40719675d1c8fb4a4811
I20260927 16:33:46.471037 4537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:46.512615 4537 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:46.513619 4537 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:46.518743 4537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: Generated new TSK 0
I20260927 16:33:46.519591 4537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:46.554304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:46.663561 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:46.663818 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:46.663887 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:46.667853 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:46.667910 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:46.668020 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:46.673249 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:46.674309 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:46.675344 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:46.685125 4549 server_base.cc:1034] running on GCE node
W20260927 16:33:46.685052 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.685052 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.685046 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:46.685820 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:46.686492 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:46.687670 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790526826687655 us; error 26 us; skew 500 ppm
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:46.690392 4549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:46.691147 4549 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:46.691237 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:46.691464 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:46.693439 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "fbb4d95cc9e841618f8285965a0dab07"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "aeec3a1073a6585766911d320b2581cc"
server_key_iv: "a14564a690a059f09ddbc41b6b19c5c1"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.694007 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb4d95cc9e841618f8285965a0dab07"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "aeec3a1073a6585766911d320b2581cc"
server_key_iv: "a14564a690a059f09ddbc41b6b19c5c1"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.697535 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 16:33:46.700009 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.701213 4549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:33:46.701345 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "fbb4d95cc9e841618f8285965a0dab07"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "aeec3a1073a6585766911d320b2581cc"
server_key_iv: "a14564a690a059f09ddbc41b6b19c5c1"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.701454 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:46.725862 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:46.728739 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:46.728945 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:46.729534 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:46.730415 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:46.730482 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.730546 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:46.730604 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.739295 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35089
I20260927 16:33:46.739315 4676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35089 every 8 connection(s)
I20260927 16:33:46.739864 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:46.740595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4549
I20260927 16:33:46.740721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:46.740973 392 external_mini_cluster.cc:1428] Setting key 84c6103a598c727d4cbb3718210fabe6
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:33:46.754128 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:46.741632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41771 (local address 127.0.98.62:37241)
0927 16:33:46.741761 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:46.741765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:46.742700 (+ 935us) server_negotiation.cc:407] Connection header received
0927 16:33:46.744054 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:46.744057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:46.744116 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:46.744224 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:46.745372 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.746290 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.747497 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.747939 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.748084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:46.750643 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:46.750650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:46.750655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:46.750690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:46.752177 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.752623 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.752626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.752627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.752698 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.753083 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.753086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.753087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.753250 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:46.753324 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:46.753893 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 16:33:46.753995 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:33:46.755394 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:46.741987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37241 (local address 127.0.98.1:41771)
0927 16:33:46.742515 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:46.742589 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:46.743852 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:46.744403 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:46.744410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:46.744765 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:46.745224 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:46.745237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.746410 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.746413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:46.747363 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:46.747372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.748083 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.748085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:46.748234 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:46.748830 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:46.748845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:46.750473 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:46.752285 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:46.752290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:46.752301 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:46.752541 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:46.752775 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:46.752777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:46.752778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:46.752996 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:46.753350 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:46.753360 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:46.753641 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 16:33:46.753847 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 16:33:46.754255 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":171,"threads_started":1}
I20260927 16:33:46.756863 4677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37241
I20260927 16:33:46.757164 4677 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:46.757743 4677 heartbeater.cc:511] Master 127.0.98.62:37241 requested a full tablet report, sending...
I20260927 16:33:46.759429 4487 ts_manager.cc:194] Registered new tserver with Master: fbb4d95cc9e841618f8285965a0dab07 (127.0.98.1:35089)
I20260927 16:33:46.760746 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:46.769172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:46.877507 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:46.877820 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:46.877869 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:46.882272 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:46.882328 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:46.882427 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:46.887741 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:46.888794 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:46.889825 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:46.899251 4682 server_base.cc:1034] running on GCE node
W20260927 16:33:46.899214 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.899231 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:46.899214 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:46.899750 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:46.900358 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:46.901508 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790526826901481 us; error 39 us; skew 500 ppm
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:46.904352 4682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:46.905055 4682 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:46.905120 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:46.905330 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:46.907382 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "eeddaecd3e314fa38ecf86e44c4b953b"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "d4e63b5098f4e01377f19b4855f58d7b"
server_key_iv: "0d7db96344ca78e8a923181c5f25960d"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.907958 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eeddaecd3e314fa38ecf86e44c4b953b"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "d4e63b5098f4e01377f19b4855f58d7b"
server_key_iv: "0d7db96344ca78e8a923181c5f25960d"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.912102 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:33:46.914558 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.915802 4682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:33:46.915934 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "eeddaecd3e314fa38ecf86e44c4b953b"
format_stamp: "Formatted at 2026-09-27 16:33:46 on dist-test-slave-t27c"
server_key: "d4e63b5098f4e01377f19b4855f58d7b"
server_key_iv: "0d7db96344ca78e8a923181c5f25960d"
server_key_version: "encryptionkey@0"
I20260927 16:33:46.916044 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:46.949039 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:46.952569 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:46.952780 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:46.953385 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:46.954283 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:46.954331 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.954396 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:46.954432 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:46.964367 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42505
I20260927 16:33:46.964383 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42505 every 8 connection(s)
I20260927 16:33:46.964989 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:46.965291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4682
I20260927 16:33:46.965409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:46.965705 392 external_mini_cluster.cc:1428] Setting key fecc117ab2deca395ddbb1627fdfa751
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 16:33:46 dist-test-slave-t27c krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:33:46.979269 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:46.966756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52137 (local address 127.0.98.62:37241)
0927 16:33:46.966881 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:46.966885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:46.968042 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 16:33:46.969711 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:46.969715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:46.969764 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:46.969849 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:46.971090 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.971624 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.972743 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.972967 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.973078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:46.975704 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:46.975710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:46.975716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:46.975750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:46.977139 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.977646 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.977653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.977658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.977748 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:46.978056 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:46.978062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:46.978066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:46.978337 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:46.978439 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:46.979050 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 16:33:46.979150 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:46.980515 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:46.967303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37241 (local address 127.0.98.2:52137)
0927 16:33:46.967863 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:46.967924 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:46.969495 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:46.970059 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:46.970066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:46.970481 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:46.970950 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:46.970963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.971747 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.971750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:46.972619 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:46.972628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:46.973120 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:46.973124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:46.973285 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:46.973891 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:46.973907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:46.975513 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:46.977247 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:46.977255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:46.977272 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:46.977526 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:46.977844 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:46.977846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:46.977847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:46.977969 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:46.978438 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:46.978442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:46.978789 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 16:33:46.979009 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 16:33:46.979397 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":121,"threads_started":1}
I20260927 16:33:46.981742 4810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37241
I20260927 16:33:46.981983 4810 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:46.982479 4810 heartbeater.cc:511] Master 127.0.98.62:37241 requested a full tablet report, sending...
I20260927 16:33:46.983534 4487 ts_manager.cc:194] Registered new tserver with Master: eeddaecd3e314fa38ecf86e44c4b953b (127.0.98.2:42505)
I20260927 16:33:46.984100 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:46.992790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:47.104907 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:47.105163 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:47.105245 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:47.109356 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:47.109421 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:47.109517 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:47.115059 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:47.116166 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:47.117249 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:47.127188 4815 server_base.cc:1034] running on GCE node
W20260927 16:33:47.127140 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.127125 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.127099 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:47.127839 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:47.128520 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:47.129689 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790526827129653 us; error 49 us; skew 500 ppm
Sep 27 16:33:47 dist-test-slave-t27c krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:47.132478 4815 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:47.133213 4815 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:47.133276 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:47.133441 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:47.135474 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "38c62f49b7886d8e713d255b5fd8bf5b"
server_key_iv: "3acf7fa5581412e15c28ba4ef2fa1a8e"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.135990 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "38c62f49b7886d8e713d255b5fd8bf5b"
server_key_iv: "3acf7fa5581412e15c28ba4ef2fa1a8e"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.139566 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:47.141773 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.142992 4815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:47.143127 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "38c62f49b7886d8e713d255b5fd8bf5b"
server_key_iv: "3acf7fa5581412e15c28ba4ef2fa1a8e"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.143234 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:47.159646 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:47.163139 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:47.163364 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:47.163985 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:47.164841 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:47.164913 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.164984 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:47.165035 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.174358 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45731
I20260927 16:33:47.174376 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45731 every 8 connection(s)
I20260927 16:33:47.174955 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:47.178750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4815
I20260927 16:33:47.178877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:47.179208 392 external_mini_cluster.cc:1428] Setting key 12ec05639da247a45b170f7175f29571
Sep 27 16:33:47 dist-test-slave-t27c krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:47.188630 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.176679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51231 (local address 127.0.98.62:37241)
0927 16:33:47.176816 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.176820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.177680 (+ 860us) server_negotiation.cc:407] Connection header received
0927 16:33:47.179221 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.179224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.179296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.179390 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:47.180886 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.181559 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.182755 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.182981 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.183115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.185395 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:47.185398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:47.185401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:47.185433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:47.186699 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:47.187189 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:47.187194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:47.187196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:47.187273 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:47.187600 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:47.187604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:47.187606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:47.187779 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:47.187887 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.188357 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.188470 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:47.189409 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.177076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37241 (local address 127.0.98.3:51231)
0927 16:33:47.177509 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.177550 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.178984 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.179600 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.179606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.180100 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:47.180749 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.180765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.181672 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.181674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.182637 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.182645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.183157 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.183161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.183331 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.183858 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:47.183873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:47.185245 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:47.186809 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:47.186815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:47.186828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:47.187089 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:47.187361 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:47.187365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:47.187367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:47.187501 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:47.187880 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:47.187886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:47.188126 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.188306 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.188624 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260927 16:33:47.190529 4943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37241
I20260927 16:33:47.190739 4943 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:47.191125 4943 heartbeater.cc:511] Master 127.0.98.62:37241 requested a full tablet report, sending...
I20260927 16:33:47.192147 4487 ts_manager.cc:194] Registered new tserver with Master: 1d0d23a8f0e74fbaa1a1df126f60a2cd (127.0.98.3:45731)
I20260927 16:33:47.192816 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51231
I20260927 16:33:47.193073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:47.203058 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.195192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.62:37241)
0927 16:33:47.195307 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.195311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.195464 (+ 153us) server_negotiation.cc:407] Connection header received
0927 16:33:47.195605 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.195608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.195659 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.195740 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:47.196746 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.197270 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.198125 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.198410 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.198531 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.200008 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:47.200012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:47.200015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:47.200049 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:47.201296 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:47.201733 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:47.201737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:47.201739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:47.201827 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:47.202101 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:47.202105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:47.202107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:47.202368 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:47.202512 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.202767 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.202888 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:33:47.205993 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:47.209002 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:47.223382 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.217366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.0.1:36342)
0927 16:33:47.217980 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.218028 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.218193 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.218989 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.218994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.219018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:47.219158 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.219165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.220480 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.220483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.221922 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.221932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.222791 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.222794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.222905 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.222939 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.223085 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.223198 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"spinlock_wait_cycles":6144,"thread_start_us":90,"threads_started":1}
I20260927 16:33:47.223443 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.217435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45731 (local address 127.0.0.1:56138)
0927 16:33:47.218012 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.218028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.218205 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.219450 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.219453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.219473 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:47.219568 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.219575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.220955 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.220958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.221969 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.221977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.223084 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.223087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.223169 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.223189 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.223259 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.223312 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"spinlock_wait_cycles":2176,"thread_start_us":103,"threads_started":1}
I20260927 16:33:47.223554 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.218311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.0.98.3:45731)
0927 16:33:47.218966 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.218978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.218997 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:33:47.219076 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.219080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.219309 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.219457 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:47.219686 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.220807 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.222094 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.222922 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.223058 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.223186 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.223284 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.223389 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":136,"threads_started":1}
I20260927 16:33:47.224042 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.217501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.98.1:35089)
0927 16:33:47.217791 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.217798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.218329 (+ 531us) server_negotiation.cc:407] Connection header received
0927 16:33:47.218410 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.218417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.218642 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.218797 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:47.219299 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.220338 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.222044 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.222673 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.222795 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.222939 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.223808 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.223900 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260927 16:33:47.224426 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.218660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.0.1:50294)
0927 16:33:47.219363 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.219378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.219457 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.220123 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.220128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.220149 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:47.220221 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.220226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.221562 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.221566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.222698 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.222708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.224027 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.224031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.224139 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.224162 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.224222 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.224288 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":98,"threads_started":1}
I20260927 16:33:47.225047 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.218548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.98.2:42505)
0927 16:33:47.218854 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.218865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.219546 (+ 681us) server_negotiation.cc:407] Connection header received
0927 16:33:47.219660 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.219666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.219839 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.219976 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:47.220341 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.221433 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.222843 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.223663 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.223774 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.224765 (+ 991us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.224807 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.224870 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":110,"threads_started":1}
I20260927 16:33:47.226614 4877 tablet_service.cc:1467] Processing CreateTablet for tablet cb04a41b6558410c9f70f57e44ecb0da (DEFAULT_TABLE table=test-table [id=11ea217793ab4d2c9c630fe8d6ee0740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:47.226788 4611 tablet_service.cc:1467] Processing CreateTablet for tablet cb04a41b6558410c9f70f57e44ecb0da (DEFAULT_TABLE table=test-table [id=11ea217793ab4d2c9c630fe8d6ee0740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:47.227717 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb04a41b6558410c9f70f57e44ecb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:47.227730 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb04a41b6558410c9f70f57e44ecb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:47.227685 4744 tablet_service.cc:1467] Processing CreateTablet for tablet cb04a41b6558410c9f70f57e44ecb0da (DEFAULT_TABLE table=test-table [id=11ea217793ab4d2c9c630fe8d6ee0740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:47.228592 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb04a41b6558410c9f70f57e44ecb0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:47.233903 4960 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: Bootstrap starting.
I20260927 16:33:47.235318 4962 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Bootstrap starting.
I20260927 16:33:47.236191 4960 tablet_bootstrap.cc:654] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:47.237089 4962 tablet_bootstrap.cc:654] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:47.237350 4960 log.cc:824] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:47.237639 4961 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Bootstrap starting.
I20260927 16:33:47.238387 4962 log.cc:824] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:47.239780 4961 tablet_bootstrap.cc:654] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:47.240103 4960 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: No bootstrap required, opened a new log
I20260927 16:33:47.240312 4960 ts_tablet_manager.cc:1397] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 16:33:47.240484 4962 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: No bootstrap required, opened a new log
I20260927 16:33:47.240675 4962 ts_tablet_manager.cc:1397] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:33:47.240952 4961 log.cc:824] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:47.242640 4961 tablet_bootstrap.cc:492] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: No bootstrap required, opened a new log
I20260927 16:33:47.242821 4961 ts_tablet_manager.cc:1397] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:33:47.244372 4960 raft_consensus.cc:348] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.244372 4962 raft_consensus.cc:348] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.244652 4962 raft_consensus.cc:374] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:47.244652 4960 raft_consensus.cc:374] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:47.244722 4960 raft_consensus.cc:729] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb4d95cc9e841618f8285965a0dab07, State: Initialized, Role: FOLLOWER
I20260927 16:33:47.244743 4962 raft_consensus.cc:729] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeddaecd3e314fa38ecf86e44c4b953b, State: Initialized, Role: FOLLOWER
I20260927 16:33:47.245268 4962 consensus_queue.cc:260] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.246081 4961 raft_consensus.cc:348] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.246364 4961 raft_consensus.cc:374] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:47.246394 4962 ts_tablet_manager.cc:1428] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 16:33:47.246423 4961 raft_consensus.cc:729] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0d23a8f0e74fbaa1a1df126f60a2cd, State: Initialized, Role: FOLLOWER
I20260927 16:33:47.246405 4810 heartbeater.cc:503] Master 127.0.98.62:37241 was elected leader, sending a full tablet report...
I20260927 16:33:47.246279 4960 consensus_queue.cc:260] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.246860 4961 consensus_queue.cc:260] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.247061 4677 heartbeater.cc:503] Master 127.0.98.62:37241 was elected leader, sending a full tablet report...
I20260927 16:33:47.247434 4960 ts_tablet_manager.cc:1428] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 16:33:47.247790 4961 ts_tablet_manager.cc:1428] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:33:47.248122 4943 heartbeater.cc:503] Master 127.0.98.62:37241 was elected leader, sending a full tablet report...
I20260927 16:33:47.276638 4966 raft_consensus.cc:482] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:47.276881 4966 raft_consensus.cc:504] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.278159 4966 leader_election.cc:288] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb4d95cc9e841618f8285965a0dab07 (127.0.98.1:35089), 1d0d23a8f0e74fbaa1a1df126f60a2cd (127.0.98.3:45731)
I20260927 16:33:47.281981 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.278445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55079 (local address 127.0.98.1:35089)
0927 16:33:47.278606 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.278611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.278628 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:47.278685 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.278691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.278748 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.278835 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:47.279157 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.279776 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.280974 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.281380 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.281461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.281542 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.281779 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.281846 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:33:47.281983 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.278633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40133 (local address 127.0.98.3:45731)
0927 16:33:47.278757 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.278760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.278950 (+ 190us) server_negotiation.cc:407] Connection header received
0927 16:33:47.279072 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.279078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.279139 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.279209 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:47.279463 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.279988 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.281025 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.281458 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.281560 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.281622 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.281810 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.281880 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:33:47.281983 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.278349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.98.2:55079)
0927 16:33:47.278482 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.278497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.278592 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.278857 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.278860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.278877 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:47.279041 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.279048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.279883 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.279886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.280842 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.280850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.281509 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.281515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.281638 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.281664 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.281732 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.281823 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":59}
I20260927 16:33:47.282269 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.278442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45731 (local address 127.0.98.2:40133)
0927 16:33:47.278851 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.278865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.278958 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.279240 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.279242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.279259 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:47.279358 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.279364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.280091 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.280094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.280875 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:47.280883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.281583 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.281586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:47.281669 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.281686 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.281778 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.281846 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260927 16:33:47.282758 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb04a41b6558410c9f70f57e44ecb0da" candidate_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb4d95cc9e841618f8285965a0dab07" is_pre_election: true
I20260927 16:33:47.282831 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb04a41b6558410c9f70f57e44ecb0da" candidate_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" is_pre_election: true
I20260927 16:33:47.283077 4897 raft_consensus.cc:2454] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeddaecd3e314fa38ecf86e44c4b953b in term 0.
I20260927 16:33:47.283077 4631 raft_consensus.cc:2454] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeddaecd3e314fa38ecf86e44c4b953b in term 0.
I20260927 16:33:47.283561 4699 leader_election.cc:302] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0d23a8f0e74fbaa1a1df126f60a2cd, eeddaecd3e314fa38ecf86e44c4b953b; no voters:
I20260927 16:33:47.283799 4966 raft_consensus.cc:2781] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:47.283896 4966 raft_consensus.cc:482] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:47.283957 4966 raft_consensus.cc:3037] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:47.285009 4966 raft_consensus.cc:504] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.285378 4966 leader_election.cc:288] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [CANDIDATE]: Term 1 election: Requested vote from peers fbb4d95cc9e841618f8285965a0dab07 (127.0.98.1:35089), 1d0d23a8f0e74fbaa1a1df126f60a2cd (127.0.98.3:45731)
I20260927 16:33:47.285763 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb04a41b6558410c9f70f57e44ecb0da" candidate_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd"
I20260927 16:33:47.285763 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb04a41b6558410c9f70f57e44ecb0da" candidate_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb4d95cc9e841618f8285965a0dab07"
I20260927 16:33:47.285897 4897 raft_consensus.cc:3037] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:47.285897 4631 raft_consensus.cc:3037] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:47.287246 4897 raft_consensus.cc:2454] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeddaecd3e314fa38ecf86e44c4b953b in term 1.
I20260927 16:33:47.287246 4631 raft_consensus.cc:2454] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeddaecd3e314fa38ecf86e44c4b953b in term 1.
I20260927 16:33:47.287600 4699 leader_election.cc:302] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0d23a8f0e74fbaa1a1df126f60a2cd, eeddaecd3e314fa38ecf86e44c4b953b; no voters:
I20260927 16:33:47.287806 4966 raft_consensus.cc:2781] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:47.288079 4966 raft_consensus.cc:686] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 1 LEADER]: Becoming Leader. State: Replica: eeddaecd3e314fa38ecf86e44c4b953b, State: Running, Role: LEADER
I20260927 16:33:47.288441 4966 consensus_queue.cc:237] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260927 16:33:47.291754 4487 catalog_manager.cc:5495] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b reported cstate change: term changed from 0 to 1, leader changed from <none> to eeddaecd3e314fa38ecf86e44c4b953b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeddaecd3e314fa38ecf86e44c4b953b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: HEALTHY } } }
I20260927 16:33:47.320806 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.317133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.0.98.2:42505)
0927 16:33:47.317297 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.317304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.317319 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:33:47.317397 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.317399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.317453 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.317571 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:47.317823 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.318358 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.319259 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:47.319484 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:47.319599 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:47.319924 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:47.320352 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:47.320497 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.320538 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.320624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:33:47.327105 4631 raft_consensus.cc:1261] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07 [term 1 FOLLOWER]: Refusing update from remote peer eeddaecd3e314fa38ecf86e44c4b953b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:47.327105 4897 raft_consensus.cc:1261] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 1 FOLLOWER]: Refusing update from remote peer eeddaecd3e314fa38ecf86e44c4b953b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:47.327903 4966 consensus_queue.cc:1035] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb4d95cc9e841618f8285965a0dab07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:47.328141 4970 consensus_queue.cc:1035] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0d23a8f0e74fbaa1a1df126f60a2cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:47.336361 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333997884724121600 to 7333997884576227328. Current Snapshot: MvccSnapshot[applied={T|T < 7333997884724121600}]
I20260927 16:33:47.344201 4486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:47.344388 4486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:47.346704 4486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d4f9fd28da44f9b94c9e5fa0a8884af: Sending DeleteTablet for 3 replicas of tablet cb04a41b6558410c9f70f57e44ecb0da
I20260927 16:33:47.347492 4877 tablet_service.cc:1514] Processing DeleteTablet for tablet cb04a41b6558410c9f70f57e44ecb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56138
I20260927 16:33:47.347492 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet cb04a41b6558410c9f70f57e44ecb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36342
I20260927 16:33:47.348012 4984 tablet_replica.cc:330] T cb04a41b6558410c9f70f57e44ecb0da P fbb4d95cc9e841618f8285965a0dab07: stopping tablet replica
I20260927 16:33:47.348160 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet cb04a41b6558410c9f70f57e44ecb0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50294
I20260927 16:33:47.348356 4983 tablet_replica.cc:330] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: stopping tablet replica
I20260927 16:33:47.348655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4549
I20260927 16:33:47.349828 4983 raft_consensus.cc:2229] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:47.349843 4985 tablet_replica.cc:330] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: stopping tablet replica
I20260927 16:33:47.350157 4985 raft_consensus.cc:2229] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:47.350422 4983 raft_consensus.cc:2258] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:47.350608 4985 raft_consensus.cc:2258] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:47.352135 4983 ts_tablet_manager.cc:1905] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:47.352136 4985 ts_tablet_manager.cc:1905] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:47.354923 4983 ts_tablet_manager.cc:1918] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:47.355104 4983 log.cc:1196] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/cb04a41b6558410c9f70f57e44ecb0da
I20260927 16:33:47.355504 4983 ts_tablet_manager.cc:1939] T cb04a41b6558410c9f70f57e44ecb0da P 1d0d23a8f0e74fbaa1a1df126f60a2cd: Deleting consensus metadata
I20260927 16:33:47.355615 4985 ts_tablet_manager.cc:1918] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:47.355719 4985 log.cc:1196] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/cb04a41b6558410c9f70f57e44ecb0da
I20260927 16:33:47.356040 4985 ts_tablet_manager.cc:1939] T cb04a41b6558410c9f70f57e44ecb0da P eeddaecd3e314fa38ecf86e44c4b953b: Deleting consensus metadata
I20260927 16:33:47.357005 4473 catalog_manager.cc:4841] TS 1d0d23a8f0e74fbaa1a1df126f60a2cd (127.0.98.3:45731): tablet cb04a41b6558410c9f70f57e44ecb0da (table test-table [id=11ea217793ab4d2c9c630fe8d6ee0740]) successfully deleted
I20260927 16:33:47.357251 4471 catalog_manager.cc:4841] TS eeddaecd3e314fa38ecf86e44c4b953b (127.0.98.2:42505): tablet cb04a41b6558410c9f70f57e44ecb0da (table test-table [id=11ea217793ab4d2c9c630fe8d6ee0740]) successfully deleted
W20260927 16:33:47.357576 4472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35089 (error 108)
I20260927 16:33:47.357760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4682
I20260927 16:33:47.359611 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.359347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.0.1:36354)
0927 16:33:47.359478 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.359540 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 16:33:47.359964 4472 catalog_manager.cc:4584] TS fbb4d95cc9e841618f8285965a0dab07 (127.0.98.1:35089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb04a41b6558410c9f70f57e44ecb0da: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35089: connect: Connection refused (error 111)
I20260927 16:33:47.363953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4815
I20260927 16:33:47.369747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4456
2026-09-27T16:33:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2287 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T16:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:47Z Disabled control of system clock
I20260927 16:33:47.399356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45729
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:33:47.508303 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:47.508581 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:47.508617 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:47.517107 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:47.517223 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:47.517258 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:47.517293 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:47.517334 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:47.524319 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:47.525509 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:47.526974 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:47.537016 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.537074 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.537474 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:47.537544 4990 server_base.cc:1034] running on GCE node
I20260927 16:33:47.538151 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:47.539374 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:47.540572 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790526827540538 us; error 46 us; skew 500 ppm
I20260927 16:33:47.542722 4990 webserver.cc:460] Webserver started at http://127.0.98.62:36015/ using document root <none> and password file <none>
I20260927 16:33:47.543421 4990 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:47.543488 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:47.543754 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:47.546000 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "5b5a76ca9d59454bbd2bf334fd0cc524"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "86781769d80a8176e7846254fa112d92"
server_key_iv: "ab7ab0acb9acf2f3f2c798c479679589"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.546693 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b5a76ca9d59454bbd2bf334fd0cc524"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "86781769d80a8176e7846254fa112d92"
server_key_iv: "ab7ab0acb9acf2f3f2c798c479679589"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.550696 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:33:47.553347 5004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.554728 4990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:47.554859 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "5b5a76ca9d59454bbd2bf334fd0cc524"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "86781769d80a8176e7846254fa112d92"
server_key_iv: "ab7ab0acb9acf2f3f2c798c479679589"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.554984 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:47.560681 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:47.561441 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:47.561635 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:47.569270 4990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45729
I20260927 16:33:47.569276 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45729 every 8 connection(s)
I20260927 16:33:47.570508 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:47.573899 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:47.574579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 4990
I20260927 16:33:47.574689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:47.575019 392 external_mini_cluster.cc:1428] Setting key ac523d43f220ab5ccdae487ed03b07b8
I20260927 16:33:47.580787 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: Bootstrap starting.
I20260927 16:33:47.581530 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.576506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.98.62:45729)
0927 16:33:47.577235 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.577255 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.577299 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:33:47.578151 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.578203 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.578643 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.579026 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:47.580092 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:47.580098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:47.580112 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:47.580123 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:47.580196 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:47.580218 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:47.580350 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.580415 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.580653 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260927 16:33:47.583235 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:47.584079 5057 log.cc:824] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:47.585826 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: No bootstrap required, opened a new log
I20260927 16:33:47.589241 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } }
I20260927 16:33:47.589497 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:47.589591 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5a76ca9d59454bbd2bf334fd0cc524, State: Initialized, Role: FOLLOWER
I20260927 16:33:47.590024 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } }
I20260927 16:33:47.590167 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:47.590250 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:47.590366 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:47.630419 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } }
I20260927 16:33:47.630940 5057 leader_election.cc:302] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5a76ca9d59454bbd2bf334fd0cc524; no voters:
I20260927 16:33:47.631335 5057 leader_election.cc:288] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:47.631609 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:47.631807 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [term 1 LEADER]: Becoming Leader. State: Replica: 5b5a76ca9d59454bbd2bf334fd0cc524, State: Running, Role: LEADER
I20260927 16:33:47.632162 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } }
I20260927 16:33:47.632897 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:47.634238 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } } }
I20260927 16:33:47.634315 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5a76ca9d59454bbd2bf334fd0cc524. Latest consensus state: current_term: 1 leader_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5a76ca9d59454bbd2bf334fd0cc524" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45729 } } }
I20260927 16:33:47.634392 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:47.634424 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:47.636030 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:47.638897 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:47.643802 5069 catalog_manager.cc:1347] Generated new cluster ID: ed9f321c26a045aba830db4927de8e39
I20260927 16:33:47.643877 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:47.662586 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:47.662940 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:47.669584 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: Generated new TSK 0
I20260927 16:33:47.670552 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:33:47.678989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:33:47.794922 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:47.795249 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:47.795339 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:47.799563 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:47.799652 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:47.799750 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:47.804793 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:47.805965 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:47.807546 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:47.817543 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.817613 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.817543 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:47.817796 5081 server_base.cc:1034] running on GCE node
I20260927 16:33:47.818235 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:47.818928 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:47.820111 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790526827820077 us; error 49 us; skew 500 ppm
I20260927 16:33:47.821872 5081 webserver.cc:460] Webserver started at http://127.0.98.1:33479/ using document root <none> and password file <none>
I20260927 16:33:47.822525 5081 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:47.822614 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:47.822844 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:47.825032 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "04540ef1fc2a4f88b0d53f729fa887dc"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "42d91a9708208c476475073198df898a"
server_key_iv: "d86d03ac9e7fb2eb05fc2bd295cca148"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.825598 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04540ef1fc2a4f88b0d53f729fa887dc"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "42d91a9708208c476475073198df898a"
server_key_iv: "d86d03ac9e7fb2eb05fc2bd295cca148"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.829321 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:33:47.831835 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.833143 5081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:47.833271 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "04540ef1fc2a4f88b0d53f729fa887dc"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "42d91a9708208c476475073198df898a"
server_key_iv: "d86d03ac9e7fb2eb05fc2bd295cca148"
server_key_version: "encryptionkey@0"
I20260927 16:33:47.833386 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:47.838611 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:47.839253 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:47.839433 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:47.840023 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:47.840894 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:47.840962 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.841035 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:47.841089 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:47.850370 5081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43703
I20260927 16:33:47.850404 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43703 every 8 connection(s)
I20260927 16:33:47.851399 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:47.855373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5081
I20260927 16:33:47.855517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:47.855794 392 external_mini_cluster.cc:1428] Setting key 68f330bd220aa66d4e5f2d1bb2f5a3a0
I20260927 16:33:47.857911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 16:33:47.858039 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.853295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38561 (local address 127.0.98.62:45729)
0927 16:33:47.853440 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:47.853445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:47.854651 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 16:33:47.856097 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:47.856100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:47.856146 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:47.856256 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:47.857565 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:47.857569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:47.857571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:47.857585 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:47.857608 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:47.857616 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:47.857706 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:47.857871 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 16:33:47.857913 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:47.858987 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:47.853772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45729 (local address 127.0.98.1:38561)
0927 16:33:47.854494 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:47.854538 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:47.855883 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:47.856431 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:47.856438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:47.856562 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:47.857264 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:47.857266 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:47.857324 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:47.857410 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:47.857704 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:47.857714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:47.857722 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:33:47.857902 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:33:47.858249 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260927 16:33:47.860513 5209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45729
I20260927 16:33:47.860786 5209 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:47.861300 5209 heartbeater.cc:511] Master 127.0.98.62:45729 requested a full tablet report, sending...
I20260927 16:33:47.863027 5021 ts_manager.cc:194] Registered new tserver with Master: 04540ef1fc2a4f88b0d53f729fa887dc (127.0.98.1:43703)
W20260927 16:33:47.972031 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:47.972313 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:47.972357 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:47.976619 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:47.976684 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:47.976785 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:47.981627 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:47.982793 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:47.984226 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:47.993886 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:47.994045 5212 server_base.cc:1034] running on GCE node
W20260927 16:33:47.993932 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:47.993923 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:47.994591 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:47.995258 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:47.996412 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790526827996394 us; error 30 us; skew 500 ppm
I20260927 16:33:47.998208 5212 webserver.cc:460] Webserver started at http://127.0.98.2:44669/ using document root <none> and password file <none>
I20260927 16:33:47.998795 5212 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:47.998855 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:47.999115 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:48.001197 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca4822e974584759adba208608166056"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "223add05702cf152848f9bf4f927858f"
server_key_iv: "58d6b8a53e7cf183f4cb3050254ab1ad"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.001765 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4822e974584759adba208608166056"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "223add05702cf152848f9bf4f927858f"
server_key_iv: "58d6b8a53e7cf183f4cb3050254ab1ad"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.005396 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:33:48.007958 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.009258 5212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:48.009385 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "ca4822e974584759adba208608166056"
format_stamp: "Formatted at 2026-09-27 16:33:47 on dist-test-slave-t27c"
server_key: "223add05702cf152848f9bf4f927858f"
server_key_iv: "58d6b8a53e7cf183f4cb3050254ab1ad"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.009505 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:48.014330 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:48.014936 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:48.015120 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:48.015655 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:48.016541 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:48.016585 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.016650 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:48.016690 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.025184 5212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34551
I20260927 16:33:48.025204 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34551 every 8 connection(s)
I20260927 16:33:48.026136 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:48.032603 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.027957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54107 (local address 127.0.98.62:45729)
0927 16:33:48.028123 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.028128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.029191 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 16:33:48.031009 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.031012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.031052 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.031129 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.032169 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.032172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.032174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.032185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.032202 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.032210 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.032284 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.032394 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.032432 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:33:48.033542 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.028385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45729 (local address 127.0.98.2:54107)
0927 16:33:48.029043 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.029083 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.030794 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.031297 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.031304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.031396 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.031924 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.031926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.031966 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.032034 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.032281 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.032287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.032291 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.032416 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.032674 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260927 16:33:48.034770 5340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45729
I20260927 16:33:48.034993 5340 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:48.035410 5340 heartbeater.cc:511] Master 127.0.98.62:45729 requested a full tablet report, sending...
I20260927 16:33:48.035758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5212
I20260927 16:33:48.035888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:48.036216 392 external_mini_cluster.cc:1428] Setting key 0810f72f5a06db78aea5b1ded30dafa5
I20260927 16:33:48.036553 5021 ts_manager.cc:194] Registered new tserver with Master: ca4822e974584759adba208608166056 (127.0.98.2:34551)
I20260927 16:33:48.038383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:33:48.153307 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:48.153662 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:48.153739 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:48.157807 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:48.157902 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:48.158007 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:48.163172 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:48.164336 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:48.165825 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:48.176222 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:48.176232 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:48.176440 5343 server_base.cc:1034] running on GCE node
W20260927 16:33:48.176479 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:48.177054 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:48.177897 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:48.179056 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790526828179037 us; error 62 us; skew 500 ppm
I20260927 16:33:48.181469 5343 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260927 16:33:48.182216 5343 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:48.182307 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:48.182546 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:48.185534 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "2f7d629e4c8b80788ffb81aa5a9f2c87"
server_key_iv: "51f93fd64fbeb2e848958993f7e9ca9a"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.186074 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "2f7d629e4c8b80788ffb81aa5a9f2c87"
server_key_iv: "51f93fd64fbeb2e848958993f7e9ca9a"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.190063 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 16:33:48.192704 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.194252 5343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:48.194386 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "2f7d629e4c8b80788ffb81aa5a9f2c87"
server_key_iv: "51f93fd64fbeb2e848958993f7e9ca9a"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.194500 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:48.199120 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:48.199738 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:48.199916 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:48.200469 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:48.201382 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:48.201452 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.201520 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:48.201575 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.210515 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34771
I20260927 16:33:48.210552 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34771 every 8 connection(s)
I20260927 16:33:48.211519 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:48.214494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5343
I20260927 16:33:48.214675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:48.214998 392 external_mini_cluster.cc:1428] Setting key 055748b466a1aa52a5d1ab8070b506ad
I20260927 16:33:48.218318 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.213578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60397 (local address 127.0.98.62:45729)
0927 16:33:48.213812 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.213817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.215002 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 16:33:48.216577 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.216580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.216625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.216713 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.217872 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.217875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.217877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.217889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.217909 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.217917 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.217991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.218116 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.218155 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 16:33:48.219215 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.214024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45729 (local address 127.0.98.3:60397)
0927 16:33:48.214822 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.214877 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.216310 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.216847 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.216854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.216946 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.217620 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.217621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.217661 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.217730 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.217994 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.217999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.218003 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.218135 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.218432 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":163,"threads_started":1}
I20260927 16:33:48.220305 5471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45729
I20260927 16:33:48.220547 5471 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:48.220948 5471 heartbeater.cc:511] Master 127.0.98.62:45729 requested a full tablet report, sending...
I20260927 16:33:48.221900 5020 ts_manager.cc:194] Registered new tserver with Master: adb1a4e6f3ce471bb3a5d77e030a2d3e (127.0.98.3:34771)
I20260927 16:33:48.229003 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:48.232359 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.230886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.0.98.62:45729)
0927 16:33:48.231027 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.231030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.231214 (+ 184us) server_negotiation.cc:407] Connection header received
0927 16:33:48.231310 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.231312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.231343 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.231411 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.232100 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.232102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.232104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.232111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.232125 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.232131 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.232191 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.232234 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.232263 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:48.235445 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:48.238432 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:48.250080 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.246530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34771 (local address 127.0.0.1:54878)
0927 16:33:48.247396 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.247449 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.247554 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.248363 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.248369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.248464 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.249118 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.249119 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.249136 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.249233 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.249732 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.249744 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.249750 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.249863 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.249936 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":79,"threads_started":1}
I20260927 16:33:48.250481 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.246622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43703 (local address 127.0.0.1:35802)
0927 16:33:48.247856 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.247870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.247940 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.248603 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.248606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.248621 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.249469 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.249470 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.249481 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.249505 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.250271 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.250277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.250281 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.250330 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.250353 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":123,"threads_started":1}
I20260927 16:33:48.250689 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.247836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.0.98.2:34551)
0927 16:33:48.248207 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.248220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.248238 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:33:48.248282 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.248287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.248443 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.248585 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.249961 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.249968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.249979 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.249990 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.250035 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.250067 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.250382 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.250435 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.250522 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":83,"threads_started":1}
I20260927 16:33:48.250808 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.246615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.98.3:34771)
0927 16:33:48.247843 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.247856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.247884 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:33:48.247942 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.247960 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.248203 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.248373 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.249399 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.249406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.249422 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.249437 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.249497 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.249540 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.249684 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.250550 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.250629 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":118,"threads_started":1}
I20260927 16:33:48.250967 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.247278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34551 (local address 127.0.0.1:44198)
0927 16:33:48.248096 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.248107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.248157 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.249262 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.249265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.249280 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.249781 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.249785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.249797 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.249825 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.250230 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.250236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.250241 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.250311 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.250347 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":91,"threads_started":1}
I20260927 16:33:48.251156 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.247681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.0.98.1:43703)
0927 16:33:48.248153 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.248165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.248194 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:33:48.248253 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.248260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.248444 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.249225 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.250068 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.250074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.250084 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.250097 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.250131 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.250152 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.250850 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.250897 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.250957 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":144,"threads_started":1}
I20260927 16:33:48.253980 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c (DEFAULT_TABLE table=test-table [id=9b04bcc0e39f4631b8de61a55790a632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:48.253980 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c (DEFAULT_TABLE table=test-table [id=9b04bcc0e39f4631b8de61a55790a632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:48.255146 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838bc8c375e440d09ce070e7d0b52e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:48.255146 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838bc8c375e440d09ce070e7d0b52e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:48.255806 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c (DEFAULT_TABLE table=test-table [id=9b04bcc0e39f4631b8de61a55790a632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:48.256543 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838bc8c375e440d09ce070e7d0b52e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:48.262475 5489 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Bootstrap starting.
I20260927 16:33:48.262907 5488 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Bootstrap starting.
I20260927 16:33:48.263279 5490 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Bootstrap starting.
I20260927 16:33:48.265321 5490 tablet_bootstrap.cc:654] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:48.265321 5488 tablet_bootstrap.cc:654] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:48.266232 5488 log.cc:824] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:48.266232 5490 log.cc:824] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:48.267518 5489 tablet_bootstrap.cc:654] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:48.268296 5489 log.cc:824] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:48.268318 5490 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: No bootstrap required, opened a new log
I20260927 16:33:48.268318 5488 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: No bootstrap required, opened a new log
I20260927 16:33:48.268524 5488 ts_tablet_manager.cc:1397] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:33:48.268524 5490 ts_tablet_manager.cc:1397] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:33:48.269966 5489 tablet_bootstrap.cc:492] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: No bootstrap required, opened a new log
I20260927 16:33:48.270215 5489 ts_tablet_manager.cc:1397] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260927 16:33:48.273137 5490 raft_consensus.cc:348] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.273137 5488 raft_consensus.cc:348] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.273223 5489 raft_consensus.cc:348] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.273412 5488 raft_consensus.cc:374] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:48.273412 5490 raft_consensus.cc:374] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:48.273454 5489 raft_consensus.cc:374] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:48.273465 5488 raft_consensus.cc:729] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4822e974584759adba208608166056, State: Initialized, Role: FOLLOWER
I20260927 16:33:48.273474 5490 raft_consensus.cc:729] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04540ef1fc2a4f88b0d53f729fa887dc, State: Initialized, Role: FOLLOWER
I20260927 16:33:48.273543 5489 raft_consensus.cc:729] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb1a4e6f3ce471bb3a5d77e030a2d3e, State: Initialized, Role: FOLLOWER
I20260927 16:33:48.273936 5490 consensus_queue.cc:260] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.273943 5488 consensus_queue.cc:260] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.273985 5489 consensus_queue.cc:260] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.274959 5490 ts_tablet_manager.cc:1428] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:33:48.274959 5488 ts_tablet_manager.cc:1428] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 16:33:48.274976 5489 ts_tablet_manager.cc:1428] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:33:48.275296 5471 heartbeater.cc:503] Master 127.0.98.62:45729 was elected leader, sending a full tablet report...
I20260927 16:33:48.275359 5209 heartbeater.cc:503] Master 127.0.98.62:45729 was elected leader, sending a full tablet report...
I20260927 16:33:48.275471 5340 heartbeater.cc:503] Master 127.0.98.62:45729 was elected leader, sending a full tablet report...
W20260927 16:33:48.277078 5341 tablet.cc:2369] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:33:48.352643 5210 tablet.cc:2369] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:48.444484 5495 raft_consensus.cc:482] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:48.444747 5495 raft_consensus.cc:504] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.445987 5495 leader_election.cc:288] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adb1a4e6f3ce471bb3a5d77e030a2d3e (127.0.98.3:34771), ca4822e974584759adba208608166056 (127.0.98.2:34551)
I20260927 16:33:48.448263 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.446347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35907 (local address 127.0.98.3:34771)
0927 16:33:48.446513 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.446518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.446758 (+ 240us) server_negotiation.cc:407] Connection header received
0927 16:33:48.446897 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.446903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.446962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.447038 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.447804 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.447807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.447810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.447828 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.447847 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.447855 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.447957 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.448068 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.448117 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:33:48.448298 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.446255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34771 (local address 127.0.98.1:35907)
0927 16:33:48.446646 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.446659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.446761 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.447095 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.447098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.447113 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.447622 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.447624 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.447633 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.447658 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.447975 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.447978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.447980 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.448053 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.448087 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":100,"threads_started":1}
I20260927 16:33:48.448379 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.446524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49209 (local address 127.0.98.2:34551)
0927 16:33:48.446649 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.446653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.446840 (+ 187us) server_negotiation.cc:407] Connection header received
0927 16:33:48.446961 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.446963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.446999 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.447101 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.447960 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.447965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.447970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.447985 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.448042 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.448054 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.448142 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.448216 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.448249 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:33:48.448426 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.446340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34551 (local address 127.0.98.1:49209)
0927 16:33:48.446705 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.446743 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.446868 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.447139 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.447142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.447162 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:48.447791 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:33:48.447793 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:48.447806 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:33:48.447836 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:48.448139 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:48.448142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:48.448145 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:33:48.448219 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 16:33:48.448248 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":78,"threads_started":1}
I20260927 16:33:48.449173 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838bc8c375e440d09ce070e7d0b52e2c" candidate_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4822e974584759adba208608166056" is_pre_election: true
I20260927 16:33:48.449172 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838bc8c375e440d09ce070e7d0b52e2c" candidate_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" is_pre_election: true
I20260927 16:33:48.449527 5294 raft_consensus.cc:2454] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04540ef1fc2a4f88b0d53f729fa887dc in term 0.
I20260927 16:33:48.449527 5425 raft_consensus.cc:2454] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04540ef1fc2a4f88b0d53f729fa887dc in term 0.
I20260927 16:33:48.450054 5099 leader_election.cc:302] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04540ef1fc2a4f88b0d53f729fa887dc, adb1a4e6f3ce471bb3a5d77e030a2d3e, ca4822e974584759adba208608166056; no voters:
I20260927 16:33:48.450402 5495 raft_consensus.cc:2781] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:48.450497 5495 raft_consensus.cc:482] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:48.450531 5495 raft_consensus.cc:3037] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:48.451601 5495 raft_consensus.cc:504] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.451963 5495 leader_election.cc:288] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [CANDIDATE]: Term 1 election: Requested vote from peers adb1a4e6f3ce471bb3a5d77e030a2d3e (127.0.98.3:34771), ca4822e974584759adba208608166056 (127.0.98.2:34551)
I20260927 16:33:48.452366 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838bc8c375e440d09ce070e7d0b52e2c" candidate_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e"
I20260927 16:33:48.452378 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838bc8c375e440d09ce070e7d0b52e2c" candidate_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4822e974584759adba208608166056"
I20260927 16:33:48.452514 5294 raft_consensus.cc:3037] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:48.452513 5425 raft_consensus.cc:3037] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:48.453778 5294 raft_consensus.cc:2454] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04540ef1fc2a4f88b0d53f729fa887dc in term 1.
I20260927 16:33:48.453778 5425 raft_consensus.cc:2454] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04540ef1fc2a4f88b0d53f729fa887dc in term 1.
I20260927 16:33:48.454073 5099 leader_election.cc:302] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04540ef1fc2a4f88b0d53f729fa887dc, adb1a4e6f3ce471bb3a5d77e030a2d3e; no voters:
I20260927 16:33:48.454280 5495 raft_consensus.cc:2781] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:48.454523 5495 raft_consensus.cc:686] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [term 1 LEADER]: Becoming Leader. State: Replica: 04540ef1fc2a4f88b0d53f729fa887dc, State: Running, Role: LEADER
I20260927 16:33:48.454877 5495 consensus_queue.cc:237] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } }
I20260927 16:33:48.458134 5019 catalog_manager.cc:5495] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 04540ef1fc2a4f88b0d53f729fa887dc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04540ef1fc2a4f88b0d53f729fa887dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 } health_report { overall_health: UNKNOWN } } }
W20260927 16:33:48.462507 5472 tablet.cc:2369] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:48.484182 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.482603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.0.98.1:43703)
0927 16:33:48.482749 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.482755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.482772 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:33:48.482865 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.482867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.482926 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.483025 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.483768 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.483771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.483774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.483785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.483805 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.483815 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.483917 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.483987 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.484028 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:33:48.490406 5425 raft_consensus.cc:1261] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 1 FOLLOWER]: Refusing update from remote peer 04540ef1fc2a4f88b0d53f729fa887dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:48.490401 5294 raft_consensus.cc:1261] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 1 FOLLOWER]: Refusing update from remote peer 04540ef1fc2a4f88b0d53f729fa887dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:48.491158 5495 consensus_queue.cc:1035] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4822e974584759adba208608166056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:48.491372 5499 consensus_queue.cc:1035] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb1a4e6f3ce471bb3a5d77e030a2d3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:48.501338 5503 mvcc.cc:205] Tried to move back new op lower bound from 7333997889488830464 to 7333997889354145792. Current Snapshot: MvccSnapshot[applied={T|T < 7333997889488830464}]
I20260927 16:33:48.501649 5502 mvcc.cc:205] Tried to move back new op lower bound from 7333997889488830464 to 7333997889354145792. Current Snapshot: MvccSnapshot[applied={T|T < 7333997889488830464 or (T in {7333997889488830464})}]
I20260927 16:33:48.506038 5021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:48.506232 5021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:48.508694 5021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b5a76ca9d59454bbd2bf334fd0cc524: Sending DeleteTablet for 3 replicas of tablet 838bc8c375e440d09ce070e7d0b52e2c
I20260927 16:33:48.509507 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:48 UTC) from {username='slave'} at 127.0.0.1:35802
I20260927 16:33:48.509645 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:48 UTC) from {username='slave'} at 127.0.0.1:44198
I20260927 16:33:48.509649 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 838bc8c375e440d09ce070e7d0b52e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:48 UTC) from {username='slave'} at 127.0.0.1:54878
I20260927 16:33:48.510233 5513 tablet_replica.cc:330] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: stopping tablet replica
I20260927 16:33:48.510228 5511 tablet_replica.cc:330] T 838bc8c375e440d09ce070e7d0b52e2c P 04540ef1fc2a4f88b0d53f729fa887dc: stopping tablet replica
I20260927 16:33:48.510433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5081
I20260927 16:33:48.512028 5513 raft_consensus.cc:2229] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:48.512391 5513 raft_consensus.cc:2258] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:48.512753 5512 tablet_replica.cc:330] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: stopping tablet replica
I20260927 16:33:48.513008 5512 raft_consensus.cc:2229] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:48.513303 5512 raft_consensus.cc:2258] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:48.514782 5513 ts_tablet_manager.cc:1905] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:48.514834 5512 ts_tablet_manager.cc:1905] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:48.517067 5513 ts_tablet_manager.cc:1918] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:48.517182 5513 log.cc:1196] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/838bc8c375e440d09ce070e7d0b52e2c
I20260927 16:33:48.517498 5513 ts_tablet_manager.cc:1939] T 838bc8c375e440d09ce070e7d0b52e2c P adb1a4e6f3ce471bb3a5d77e030a2d3e: Deleting consensus metadata
W20260927 16:33:48.517937 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43703 (error 108)
I20260927 16:33:48.518133 5512 ts_tablet_manager.cc:1918] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:48.518246 5512 log.cc:1196] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/838bc8c375e440d09ce070e7d0b52e2c
I20260927 16:33:48.518548 5512 ts_tablet_manager.cc:1939] T 838bc8c375e440d09ce070e7d0b52e2c P ca4822e974584759adba208608166056: Deleting consensus metadata
I20260927 16:33:48.518779 5008 catalog_manager.cc:4841] TS adb1a4e6f3ce471bb3a5d77e030a2d3e (127.0.98.3:34771): tablet 838bc8c375e440d09ce070e7d0b52e2c (table test-table [id=9b04bcc0e39f4631b8de61a55790a632]) successfully deleted
I20260927 16:33:48.518841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5212
I20260927 16:33:48.520982 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.520621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43703 (local address 127.0.0.1:35808)
0927 16:33:48.520783 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.520866 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 16:33:48.521121 5005 catalog_manager.cc:4584] TS 04540ef1fc2a4f88b0d53f729fa887dc (127.0.98.1:43703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 838bc8c375e440d09ce070e7d0b52e2c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43703: connect: Connection refused (error 111)
I20260927 16:33:48.524813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5343
I20260927 16:33:48.530898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 4990
I20260927 16:33:48.530934 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.524897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34551 (local address 127.0.0.1:44200)
0927 16:33:48.530774 (+ 5877us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.530833 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5778}
2026-09-27T16:33:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1160 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 16:33:48.552388 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:33:48.553974 392 webserver.cc:460] Webserver started at https://127.0.0.1:36777/ using document root <none> and password file <none>
I20260927 16:33:48.554209 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:33:48.554502 392 webserver.cc:460] Webserver started at http://127.0.0.1:42049/ using document root <none> and password file <none>
2026-09-27T16:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:48Z Disabled control of system clock
I20260927 16:33:48.563431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33599
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33599
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36777/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/testchainca.cert with env {}
W20260927 16:33:48.675312 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:48.675598 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:48.675637 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:48.679762 5520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:48.679827 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:48.679843 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:48.679859 5520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:48.679874 5520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:33:48.679891 5520 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:33:48.679904 5520 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36777/jwks.json
I20260927 16:33:48.685287 5520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36777/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:48.686383 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:48.687417 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:48.696596 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:48.696676 5520 server_base.cc:1034] running on GCE node
W20260927 16:33:48.696617 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:48.696617 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:48.697539 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:48.698879 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:48.700052 5520 hybrid_clock.cc:648] HybridClock initialized: now 1790526828700020 us; error 49 us; skew 500 ppm
I20260927 16:33:48.702072 5520 webserver.cc:460] Webserver started at http://127.0.98.62:43823/ using document root <none> and password file <none>
I20260927 16:33:48.702711 5520 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:48.702777 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:48.703051 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:48.705170 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "b97759d94cf19abffce1d842ac7b7277"
server_key_iv: "b1cf32720179c1a5400191beeb87a9a5"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.705786 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "b97759d94cf19abffce1d842ac7b7277"
server_key_iv: "b1cf32720179c1a5400191beeb87a9a5"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.709640 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:33:48.712111 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:48.713353 5520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:48.713517 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5"
format_stamp: "Formatted at 2026-09-27 16:33:48 on dist-test-slave-t27c"
server_key: "b97759d94cf19abffce1d842ac7b7277"
server_key_iv: "b1cf32720179c1a5400191beeb87a9a5"
server_key_version: "encryptionkey@0"
I20260927 16:33:48.713636 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:48.734455 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:48.735221 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:48.735450 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:48.742661 5520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33599
I20260927 16:33:48.742669 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33599 every 8 connection(s)
I20260927 16:33:48.743665 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:48.746932 5587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:48.749261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5520
I20260927 16:33:48.749394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:48.749644 392 external_mini_cluster.cc:1428] Setting key 935d73f366dbb095d6cbf2688651585d
I20260927 16:33:48.752693 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5: Bootstrap starting.
I20260927 16:33:48.755623 5587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:48.756458 5587 log.cc:824] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:48.758598 5587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5: No bootstrap required, opened a new log
I20260927 16:33:48.759191 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.751075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.0.98.62:33599)
0927 16:33:48.751678 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.751693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.751730 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:33:48.752329 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.752345 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.752751 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.753083 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:48.753437 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.754479 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.755751 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.756252 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.756373 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:48.757310 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:48.757316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:48.757338 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:48.757738 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:48.757795 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:48.757815 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:48.757928 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:48.758086 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:33:48.758382 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":130,"threads_started":1}
I20260927 16:33:48.762136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:33:48.762945 5587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } }
I20260927 16:33:48.763276 5587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:48.763355 5587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a55bd3d9704b7cb3056e5d2d6d60c5, State: Initialized, Role: FOLLOWER
I20260927 16:33:48.763870 5587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } }
I20260927 16:33:48.764086 5587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:48.764161 5587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:48.764268 5587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:48.765308 5587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } }
I20260927 16:33:48.765683 5587 leader_election.cc:302] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39a55bd3d9704b7cb3056e5d2d6d60c5; no voters:
I20260927 16:33:48.766074 5587 leader_election.cc:288] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:48.766206 5592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:48.766450 5592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [term 1 LEADER]: Becoming Leader. State: Replica: 39a55bd3d9704b7cb3056e5d2d6d60c5, State: Running, Role: LEADER
I20260927 16:33:48.766782 5592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } }
I20260927 16:33:48.767510 5587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:48.768615 5594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a55bd3d9704b7cb3056e5d2d6d60c5. Latest consensus state: current_term: 1 leader_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } } }
I20260927 16:33:48.768800 5594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:48.769179 5601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:48.769423 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a55bd3d9704b7cb3056e5d2d6d60c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33599 } } }
I20260927 16:33:48.769578 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:48.772696 5601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:48.778940 5601 catalog_manager.cc:1347] Generated new cluster ID: 8fbacd4c2fd447c7b1a8ff168de6d56b
I20260927 16:33:48.779027 5601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:48.787847 5601 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:48.789187 5601 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:48.797761 5601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39a55bd3d9704b7cb3056e5d2d6d60c5: Generated new TSK 0
I20260927 16:33:48.798594 5601 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:33:48.839311 5617 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 16:33:48.839433 5617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:33:48.839924 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:48.812536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.0.98.62:33599)
0927 16:33:48.812678 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:48.812688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:48.812883 (+ 195us) server_negotiation.cc:407] Connection header received
0927 16:33:48.813120 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:48.813123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:48.813172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:48.813261 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:33:48.813577 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.814112 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.815483 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.815703 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.815780 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:48.816178 (+ 398us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:33:48.839461 (+ 23283us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:33:48.839772 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 16:33:48.840032 5590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 16:33:48.841158 5616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:48.812376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33599 (local address 127.0.0.1:59616)
0927 16:33:48.812775 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:48.812802 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:48.812951 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:48.813310 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:48.813314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:48.813323 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:33:48.813454 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:48.813461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.814241 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.814245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:48.815362 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:48.815375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:48.815815 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:48.815838 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:48.815934 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:48.815991 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:33:48.839711 (+ 23720us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:33:48.839991 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36777/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 16:33:48.842024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5520
2026-09-27T16:33:48Z chronyd exiting
I20260927 16:33:48.879702 392 test_util.cc:175] -----------------------------------------------
I20260927 16:33:48.879810 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526802550915-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 16:33:48.880864 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:33:48.881747 392 webserver.cc:460] Webserver started at https://127.0.0.1:35805/ using document root <none> and password file <none>
I20260927 16:33:48.881881 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:33:48.882216 392 webserver.cc:460] Webserver started at http://127.0.0.1:41693/ using document root <none> and password file <none>
2026-09-27T16:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:48Z Disabled control of system clock
I20260927 16:33:48.891076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33693
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35805/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/testchainca.cert with env {}
W20260927 16:33:49.001552 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:49.001837 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:49.001879 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:49.005975 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:49.006043 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:49.006059 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:49.006076 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:49.006091 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:33:49.006142 5623 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:33:49.006192 5623 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35805/jwks.json
I20260927 16:33:49.011799 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35805/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:49.013010 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:49.014098 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:49.022691 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:49.022745 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:49.022938 5623 server_base.cc:1034] running on GCE node
W20260927 16:33:49.022698 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:49.023593 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:49.024756 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:49.025925 5623 hybrid_clock.cc:648] HybridClock initialized: now 1790526829025913 us; error 25 us; skew 500 ppm
I20260927 16:33:49.027854 5623 webserver.cc:460] Webserver started at http://127.0.98.62:44813/ using document root <none> and password file <none>
I20260927 16:33:49.028450 5623 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:49.028514 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:49.028746 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:49.030920 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "edf2b2e39657442194a1ed241819c94b"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "5eba4e9264a098f0c5ff0ad5bd3de619"
server_key_iv: "0f90b1aeaa2818562d11513eea6cad6a"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.031510 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "edf2b2e39657442194a1ed241819c94b"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "5eba4e9264a098f0c5ff0ad5bd3de619"
server_key_iv: "0f90b1aeaa2818562d11513eea6cad6a"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.035209 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:49.037628 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:49.038856 5623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:49.039008 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "edf2b2e39657442194a1ed241819c94b"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "5eba4e9264a098f0c5ff0ad5bd3de619"
server_key_iv: "0f90b1aeaa2818562d11513eea6cad6a"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.039125 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:49.081620 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:49.082492 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:49.082696 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:49.089745 5623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33693
I20260927 16:33:49.089746 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33693 every 8 connection(s)
I20260927 16:33:49.090853 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:49.094065 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:49.096976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5623
I20260927 16:33:49.097115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:49.097374 392 external_mini_cluster.cc:1428] Setting key 749064b84e8ab2daefd520ff9717cc33
I20260927 16:33:49.100062 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b: Bootstrap starting.
I20260927 16:33:49.102808 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:49.103828 5690 log.cc:824] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:49.106436 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b: No bootstrap required, opened a new log
I20260927 16:33:49.108278 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.098869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.62:33693)
0927 16:33:49.099700 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.099726 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.099765 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:33:49.100489 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.100507 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.100929 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.101299 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:49.101740 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.103300 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.104625 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.105259 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.105444 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.106334 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:49.106340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:49.106363 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:49.106825 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:49.106889 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:49.106909 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:49.107019 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:49.107156 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 16:33:49.107545 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260927 16:33:49.110268 5690 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260927 16:33:49.110560 5690 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:49.110628 5690 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf2b2e39657442194a1ed241819c94b, State: Initialized, Role: FOLLOWER
I20260927 16:33:49.111104 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260927 16:33:49.111265 5690 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:49.111331 5690 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:49.111344 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:33:49.111433 5690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:49.112589 5690 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260927 16:33:49.112936 5690 leader_election.cc:302] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf2b2e39657442194a1ed241819c94b; no voters:
I20260927 16:33:49.113248 5690 leader_election.cc:288] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:49.113349 5695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:49.113519 5695 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [term 1 LEADER]: Becoming Leader. State: Replica: edf2b2e39657442194a1ed241819c94b, State: Running, Role: LEADER
I20260927 16:33:49.113807 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } }
I20260927 16:33:49.114547 5690 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:49.115543 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf2b2e39657442194a1ed241819c94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } } }
I20260927 16:33:49.115679 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:49.115983 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf2b2e39657442194a1ed241819c94b. Latest consensus state: current_term: 1 leader_uuid: "edf2b2e39657442194a1ed241819c94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf2b2e39657442194a1ed241819c94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33693 } } }
I20260927 16:33:49.116091 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:49.116070 5704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:49.119197 5704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:49.124735 5704 catalog_manager.cc:1347] Generated new cluster ID: 79b49868a7fa4ce7bb850d63acb6707c
I20260927 16:33:49.124804 5704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:49.140702 5704 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:49.142009 5704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:49.147255 5704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf2b2e39657442194a1ed241819c94b: Generated new TSK 0
I20260927 16:33:49.148064 5704 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:33:49.181332 5720 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 16:33:49.181437 5720 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:33:49.182117 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.166143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.0.98.62:33693)
0927 16:33:49.166296 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.166301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.166455 (+ 154us) server_negotiation.cc:407] Connection header received
0927 16:33:49.166622 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.166624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.166671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.166761 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:33:49.167023 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.167549 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.168769 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.169000 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.169074 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.169407 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:33:49.181760 (+ 12353us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:33:49.181997 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44956: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 16:33:49.182265 5693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44956: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 16:33:49.182394 5719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:49.166001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33693 (local address 127.0.0.1:44956)
0927 16:33:49.166352 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:49.166368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:49.166490 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:49.166786 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:49.166789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:49.166798 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:33:49.166912 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.166919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.167655 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.167658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.168640 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.168650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.169093 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.169115 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.169206 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.169236 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:33:49.181975 (+ 12739us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:33:49.182212 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":135,"threads_started":1}
I20260927 16:33:49.183079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5623
2026-09-27T16:33:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 16:33:49.222361 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:33:49.223219 392 webserver.cc:460] Webserver started at https://127.0.0.1:40661/ using document root <none> and password file <none>
I20260927 16:33:49.223311 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:33:49.223551 392 webserver.cc:460] Webserver started at http://127.0.0.1:46517/ using document root <none> and password file <none>
2026-09-27T16:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:49Z Disabled control of system clock
I20260927 16:33:49.232476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32993
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40661/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:33:49.342850 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:49.343137 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:49.343199 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:49.347333 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:49.347397 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:49.347414 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:49.347430 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:49.347445 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:33:49.347461 5726 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:33:49.347512 5726 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40661/jwks.json
I20260927 16:33:49.352735 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40661/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:49.353938 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:49.355412 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:49.363986 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:49.364051 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:49.363988 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:49.364122 5726 server_base.cc:1034] running on GCE node
I20260927 16:33:49.364698 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:49.365830 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:49.367002 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790526829366988 us; error 25 us; skew 500 ppm
I20260927 16:33:49.368813 5726 webserver.cc:460] Webserver started at http://127.0.98.62:44361/ using document root <none> and password file <none>
I20260927 16:33:49.369423 5726 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:49.369488 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:49.369763 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:49.372076 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "946fd95116ed4618957babb144cb1bfd"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "f4964aa1eca94677e205ddd9bcf3bc21"
server_key_iv: "232e85c14fbafca5cc48888a55615bc9"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.372696 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "946fd95116ed4618957babb144cb1bfd"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "f4964aa1eca94677e205ddd9bcf3bc21"
server_key_iv: "232e85c14fbafca5cc48888a55615bc9"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.376569 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:33:49.379134 5740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:49.380409 5726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:49.380546 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "946fd95116ed4618957babb144cb1bfd"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "f4964aa1eca94677e205ddd9bcf3bc21"
server_key_iv: "232e85c14fbafca5cc48888a55615bc9"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.380666 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:49.405067 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:49.405808 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:49.406023 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:49.413347 5726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32993
I20260927 16:33:49.413376 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32993 every 8 connection(s)
I20260927 16:33:49.414388 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:49.417629 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:49.418437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5726
I20260927 16:33:49.418550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:49.418843 392 external_mini_cluster.cc:1428] Setting key debc608bc6836c5dc82ff7f396d9960b
I20260927 16:33:49.423864 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: Bootstrap starting.
I20260927 16:33:49.426637 5793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:49.427448 5793 log.cc:824] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:49.428503 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.420254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.62:32993)
0927 16:33:49.420897 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.420911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.420945 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:33:49.421583 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.421599 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.421992 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.422379 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:49.422847 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.423930 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.425136 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.425588 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.425707 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.426615 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:49.426620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:49.426632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:49.427071 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:49.427124 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:49.427143 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:49.427217 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:49.427396 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:33:49.427671 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260927 16:33:49.429474 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: No bootstrap required, opened a new log
I20260927 16:33:49.431375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:33:49.433745 5793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260927 16:33:49.433960 5793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:49.433995 5793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946fd95116ed4618957babb144cb1bfd, State: Initialized, Role: FOLLOWER
I20260927 16:33:49.434401 5793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260927 16:33:49.434520 5793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:49.434566 5793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:49.434634 5793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:49.435462 5793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260927 16:33:49.435750 5793 leader_election.cc:302] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 946fd95116ed4618957babb144cb1bfd; no voters:
I20260927 16:33:49.436012 5793 leader_election.cc:288] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:49.436139 5798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:49.436391 5798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [term 1 LEADER]: Becoming Leader. State: Replica: 946fd95116ed4618957babb144cb1bfd, State: Running, Role: LEADER
I20260927 16:33:49.436735 5798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } }
I20260927 16:33:49.437152 5793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:49.438330 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 946fd95116ed4618957babb144cb1bfd. Latest consensus state: current_term: 1 leader_uuid: "946fd95116ed4618957babb144cb1bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } }
I20260927 16:33:49.438465 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:49.439318 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "946fd95116ed4618957babb144cb1bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946fd95116ed4618957babb144cb1bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } }
I20260927 16:33:49.439486 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd [sys.catalog]: This master's current role is: LEADER
W20260927 16:33:49.441748 5813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:49.441831 5813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:49.441918 5807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:49.442643 5807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:49.447537 5807 catalog_manager.cc:1347] Generated new cluster ID: d74dbb99bcff4b64a17b6dcc1ae868b4
I20260927 16:33:49.447682 5807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:49.459873 5807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:49.461164 5807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:49.472632 5807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 946fd95116ed4618957babb144cb1bfd: Generated new TSK 0
I20260927 16:33:49.473507 5807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:33:49.486613 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.481443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.62:32993)
0927 16:33:49.481580 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.481584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.481777 (+ 193us) server_negotiation.cc:407] Connection header received
0927 16:33:49.481944 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.481946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.481992 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.482080 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:33:49.482358 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.482887 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.484122 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.484380 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.484458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.484804 (+ 346us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:33:49.486212 (+ 1408us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40661/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:33:49.486467 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40661/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 16:33:49.486681 5796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40661/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 16:33:49.488565 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:49.481383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.0.1:42356)
0927 16:33:49.481677 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:49.481691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:49.481812 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:49.482103 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:49.482106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:49.482114 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:33:49.482251 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.482257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.482996 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.483000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.484014 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.484025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.484477 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.484501 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.484599 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.484628 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:33:49.486390 (+ 1762us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40661/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:33:49.488381 (+ 1991us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32993: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40661/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":1707,"thread_start_us":116,"threads_started":1}
I20260927 16:33:49.489217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5726
E20260927 16:33:49.498056 5823 webserver.cc:533] Webserver: sslize failed:
E20260927 16:33:49.498131 5823 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T16:33:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T16:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:49Z Disabled control of system clock
I20260927 16:33:49.537520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40917
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40917
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/testchainca.cert with env {}
W20260927 16:33:49.649665 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:49.649974 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:49.650017 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:49.654273 5827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:49.654348 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:49.654364 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:49.654382 5827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:49.654397 5827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:33:49.654452 5827 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:33:49.654482 5827 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 16:33:49.660059 5827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:49.661233 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:49.662329 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:49.670773 5827 server_base.cc:1034] running on GCE node
W20260927 16:33:49.670725 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:49.670760 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:49.670725 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:49.671586 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:49.672698 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:49.673895 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790526829673857 us; error 50 us; skew 500 ppm
I20260927 16:33:49.675844 5827 webserver.cc:460] Webserver started at http://127.0.98.62:43137/ using document root <none> and password file <none>
I20260927 16:33:49.676434 5827 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:49.676527 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:49.676729 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:49.678803 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "a292ba21d38a4f80aba9b35dba1233da"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "aad8671a7cc98f5d91846f1380ce0963"
server_key_iv: "b637795bbfd667b157cfa4f4bc1239cd"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.679407 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "a292ba21d38a4f80aba9b35dba1233da"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "aad8671a7cc98f5d91846f1380ce0963"
server_key_iv: "b637795bbfd667b157cfa4f4bc1239cd"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.683049 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:49.685423 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:49.686679 5827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:49.686816 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "a292ba21d38a4f80aba9b35dba1233da"
format_stamp: "Formatted at 2026-09-27 16:33:49 on dist-test-slave-t27c"
server_key: "aad8671a7cc98f5d91846f1380ce0963"
server_key_iv: "b637795bbfd667b157cfa4f4bc1239cd"
server_key_version: "encryptionkey@0"
I20260927 16:33:49.686935 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:49.702071 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:49.702844 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:49.703048 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:49.709785 5827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40917
I20260927 16:33:49.709821 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40917 every 8 connection(s)
I20260927 16:33:49.710832 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:49.713181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5827
I20260927 16:33:49.713305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:49.713584 392 external_mini_cluster.cc:1428] Setting key 80f24d3056e3a577bbae4539aae42349
I20260927 16:33:49.713752 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:49.720938 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da: Bootstrap starting.
I20260927 16:33:49.722607 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.714977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.0.98.62:40917)
0927 16:33:49.715542 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.715560 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.715594 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:33:49.716214 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.716229 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.716603 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.716886 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:49.717219 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.718155 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.719341 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.719789 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.719908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.720787 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:49.720792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:49.720804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:33:49.721249 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:49.721304 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:33:49.721330 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:49.721403 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:49.721569 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:33:49.721871 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 16:33:49.723393 5894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:49.724128 5894 log.cc:824] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:49.725289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:33:49.726266 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da: No bootstrap required, opened a new log
I20260927 16:33:49.729905 5894 raft_consensus.cc:348] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260927 16:33:49.730161 5894 raft_consensus.cc:374] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:49.730263 5894 raft_consensus.cc:729] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a292ba21d38a4f80aba9b35dba1233da, State: Initialized, Role: FOLLOWER
I20260927 16:33:49.730748 5894 consensus_queue.cc:260] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260927 16:33:49.730912 5894 raft_consensus.cc:388] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:49.731009 5894 raft_consensus.cc:482] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:49.731114 5894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:49.732110 5894 raft_consensus.cc:504] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260927 16:33:49.732482 5894 leader_election.cc:302] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a292ba21d38a4f80aba9b35dba1233da; no voters:
I20260927 16:33:49.732841 5894 leader_election.cc:288] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:49.732951 5899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:49.733192 5899 raft_consensus.cc:686] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [term 1 LEADER]: Becoming Leader. State: Replica: a292ba21d38a4f80aba9b35dba1233da, State: Running, Role: LEADER
I20260927 16:33:49.733517 5899 consensus_queue.cc:237] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260927 16:33:49.734163 5894 sys_catalog.cc:564] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:49.735321 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a292ba21d38a4f80aba9b35dba1233da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } } }
I20260927 16:33:49.735453 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:49.735816 5908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:49.735792 5901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [sys.catalog]: SysCatalogTable state changed. Reason: New leader a292ba21d38a4f80aba9b35dba1233da. Latest consensus state: current_term: 1 leader_uuid: "a292ba21d38a4f80aba9b35dba1233da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a292ba21d38a4f80aba9b35dba1233da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } } }
I20260927 16:33:49.736627 5901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:49.738596 5908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:49.744376 5908 catalog_manager.cc:1347] Generated new cluster ID: c54ad7e1ef2d4ef195051fa2d707195c
I20260927 16:33:49.744457 5908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:49.762539 5908 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:49.763552 5908 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:49.769985 5908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a292ba21d38a4f80aba9b35dba1233da: Generated new TSK 0
I20260927 16:33:49.770797 5908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:33:49.875372 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:49.781080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.98.62:40917)
0927 16:33:49.781215 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:49.781220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:49.781408 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:33:49.781618 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:49.781620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:49.781672 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:49.781788 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:33:49.782051 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.782646 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.783542 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.783772 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.783891 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.784215 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:33:49.874884 (+ 90669us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:33:49.875207 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39284: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:33:49.875456 5897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39284: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 16:33:49.875525 5923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:49.780978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40917 (local address 127.0.0.1:39284)
0927 16:33:49.781310 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:49.781324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:49.781460 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:49.781798 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:49.781802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:49.781810 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:33:49.781942 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.781947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.782762 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.782766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.783418 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:49.783426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:49.783880 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:49.783907 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:49.783996 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:49.784029 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:33:49.875138 (+ 91109us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:33:49.875321 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260927 16:33:49.876318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5827
2026-09-27T16:33:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (368 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:50 dist-test-slave-t27c krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:50 dist-test-slave-t27c krb5kdc[5935](info): set up 2 sockets
Sep 27 16:33:50 dist-test-slave-t27c krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260927 16:33:51.170532 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:33:51.291360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39471
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:51.401656 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:51.401928 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:51.401968 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:51.406052 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:51.406128 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:51.406147 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:51.406219 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:51.406255 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:51.411789 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:51.413025 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:51.414193 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:51.422863 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:51.422894 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:51.422948 5948 server_base.cc:1034] running on GCE node
W20260927 16:33:51.422863 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:51.423759 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:51.424880 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:51.426065 5948 hybrid_clock.cc:648] HybridClock initialized: now 1790526831426034 us; error 44 us; skew 500 ppm
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:51.434023 5948 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:33:51.435119 5948 webserver.cc:460] Webserver started at http://127.0.98.62:36437/ using document root <none> and password file <none>
I20260927 16:33:51.435721 5948 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:51.435815 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:51.436055 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:51.438339 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "2e9fcac79087449d829c8f63fc109dc0"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "1d833c04bad3d79f8db6ccac0dec0372"
server_key_iv: "469cfd7ca958e32aaeefb4f318a05fc1"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.438926 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e9fcac79087449d829c8f63fc109dc0"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "1d833c04bad3d79f8db6ccac0dec0372"
server_key_iv: "469cfd7ca958e32aaeefb4f318a05fc1"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.442659 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:51.445048 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.446302 5948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:51.446436 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "2e9fcac79087449d829c8f63fc109dc0"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "1d833c04bad3d79f8db6ccac0dec0372"
server_key_iv: "469cfd7ca958e32aaeefb4f318a05fc1"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.446550 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:51.460494 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:51.463527 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:51.463743 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:51.470700 5948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39471
I20260927 16:33:51.470989 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39471 every 8 connection(s)
I20260927 16:33:51.471721 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:51.474406 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:51.477988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 5948
I20260927 16:33:51.478083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:51.478369 392 external_mini_cluster.cc:1428] Setting key 37a9162e90f9fdb5a79ce68627c62958
I20260927 16:33:51.480039 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0: Bootstrap starting.
I20260927 16:33:51.482414 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:51.483135 6016 log.cc:824] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:51.485102 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0: No bootstrap required, opened a new log
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:51.489001 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } }
I20260927 16:33:51.489249 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:51.489327 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9fcac79087449d829c8f63fc109dc0, State: Initialized, Role: FOLLOWER
I20260927 16:33:51.489764 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } }
I20260927 16:33:51.489912 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:51.489980 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:51.490097 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:51.490985 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } }
I20260927 16:33:51.491331 6016 leader_election.cc:302] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e9fcac79087449d829c8f63fc109dc0; no voters:
I20260927 16:33:51.491679 6016 leader_election.cc:288] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:51.491984 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:51.492208 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e9fcac79087449d829c8f63fc109dc0, State: Running, Role: LEADER
I20260927 16:33:51.492558 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } }
I20260927 16:33:51.493054 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:51.479766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.0.98.62:39471)
0927 16:33:51.480164 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:51.480193 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:51.480241 (+ 48us) server_negotiation.cc:407] Connection header received
0927 16:33:51.480844 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:51.480860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:51.481175 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:51.481462 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:51.482336 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.483419 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.484514 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.484966 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.485084 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:51.487860 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:51.487865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:51.487878 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:51.487903 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:51.489804 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.490366 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.490372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.490379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.490494 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.490794 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.490797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.490799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.491262 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:51.491475 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:51.491772 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 16:33:51.492096 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260927 16:33:51.493052 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:51.494433 6023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e9fcac79087449d829c8f63fc109dc0. Latest consensus state: current_term: 1 leader_uuid: "2e9fcac79087449d829c8f63fc109dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } } }
I20260927 16:33:51.494457 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e9fcac79087449d829c8f63fc109dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fcac79087449d829c8f63fc109dc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39471 } } }
I20260927 16:33:51.494552 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:51.494550 6023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:51.495872 6030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:51.498595 6030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:51.503497 6030 catalog_manager.cc:1347] Generated new cluster ID: 52aed2df49844fe08cec4907ffd255ce
I20260927 16:33:51.503561 6030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:51.518630 6030 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:51.519670 6030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:51.524643 6030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e9fcac79087449d829c8f63fc109dc0: Generated new TSK 0
I20260927 16:33:51.525313 6030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:33:51.555555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:51.670082 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:51.670406 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:51.670488 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:51.674595 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:51.674659 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:51.674738 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:51.680060 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:51.681187 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:51.682296 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:51.692183 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:51.692166 6042 server_base.cc:1034] running on GCE node
W20260927 16:33:51.692224 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:51.692162 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:51.692937 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:51.693822 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:51.695005 6042 hybrid_clock.cc:648] HybridClock initialized: now 1790526831694953 us; error 68 us; skew 500 ppm
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:51.697788 6042 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:33:51.698776 6042 webserver.cc:460] Webserver started at http://127.0.98.1:43109/ using document root <none> and password file <none>
I20260927 16:33:51.699404 6042 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:51.699509 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:51.699753 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:51.701879 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c666456c9974c6fbf1c7db6a87de0d1"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "f1b9c4fd8673a262e1b344148ae122fd"
server_key_iv: "82f306222218895a4a1ba642f95beeed"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.702497 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c666456c9974c6fbf1c7db6a87de0d1"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "f1b9c4fd8673a262e1b344148ae122fd"
server_key_iv: "82f306222218895a4a1ba642f95beeed"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.706254 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:33:51.708642 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.709861 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:51.709998 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "0c666456c9974c6fbf1c7db6a87de0d1"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "f1b9c4fd8673a262e1b344148ae122fd"
server_key_iv: "82f306222218895a4a1ba642f95beeed"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.710108 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:51.736553 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:51.739607 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:51.739838 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:51.740469 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:51.741317 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:51.741386 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.741457 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:51.741498 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.750526 6042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43369
I20260927 16:33:51.750567 6170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43369 every 8 connection(s)
I20260927 16:33:51.751509 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:51.752089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6042
I20260927 16:33:51.752203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:51.752504 392 external_mini_cluster.cc:1428] Setting key db93eed7ac598848cb996e3ea0cb08d7
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:51.768592 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:51.753453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45663 (local address 127.0.98.62:39471)
0927 16:33:51.753591 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:51.753595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:51.754721 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 16:33:51.756482 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:51.756485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:51.756538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:51.756621 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:51.757813 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.758426 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.759566 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.759803 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.759917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:51.764390 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:51.764394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:51.764396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:51.764422 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:51.766124 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.766838 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.766840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.766842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.766917 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.767379 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.767382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.767383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.767563 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:51.767639 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:51.768349 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 16:33:51.768463 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:51.769973 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:51.753983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39471 (local address 127.0.98.1:45663)
0927 16:33:51.754557 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:51.754616 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:51.756266 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:51.756806 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:51.756813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:51.757171 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:51.757669 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:51.757692 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.758547 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.758551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:51.759438 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:51.759447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.759955 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.759960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:51.760129 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:51.760740 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:51.760755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:51.764204 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:51.766291 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:51.766301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:51.766322 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:51.766737 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:51.767066 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:51.767072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:51.767074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:51.767283 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:51.767680 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:51.767690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:51.768049 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 16:33:51.768297 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 16:33:51.768700 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":173,"threads_started":1}
I20260927 16:33:51.771510 6171 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39471
I20260927 16:33:51.771792 6171 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:51.772357 6171 heartbeater.cc:511] Master 127.0.98.62:39471 requested a full tablet report, sending...
I20260927 16:33:51.773960 5980 ts_manager.cc:194] Registered new tserver with Master: 0c666456c9974c6fbf1c7db6a87de0d1 (127.0.98.1:43369)
I20260927 16:33:51.775195 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:33:51.781296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:51.892285 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:51.892592 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:51.892663 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:51.897128 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:51.897222 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:51.897311 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:51.902675 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:51.903769 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:51.904798 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:51.914548 6181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:51.914535 6184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:51.914563 6182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:51.914903 6176 server_base.cc:1034] running on GCE node
I20260927 16:33:51.915273 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:51.915848 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:51.917002 6176 hybrid_clock.cc:648] HybridClock initialized: now 1790526831916968 us; error 47 us; skew 500 ppm
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:51.919670 6176 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:33:51.920519 6176 webserver.cc:460] Webserver started at http://127.0.98.2:42335/ using document root <none> and password file <none>
I20260927 16:33:51.921101 6176 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:51.921161 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:51.921398 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:51.923485 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4622c57e86d4b5e954a3b8d403c44bb"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "ef0c31c58f0d1a0c6a6435eabbef7790"
server_key_iv: "01545c73cc3db3befbf31b11f5bc45a8"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.924058 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4622c57e86d4b5e954a3b8d403c44bb"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "ef0c31c58f0d1a0c6a6435eabbef7790"
server_key_iv: "01545c73cc3db3befbf31b11f5bc45a8"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.927764 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:33:51.930099 6191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.931526 6176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:51.931656 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "d4622c57e86d4b5e954a3b8d403c44bb"
format_stamp: "Formatted at 2026-09-27 16:33:51 on dist-test-slave-t27c"
server_key: "ef0c31c58f0d1a0c6a6435eabbef7790"
server_key_iv: "01545c73cc3db3befbf31b11f5bc45a8"
server_key_version: "encryptionkey@0"
I20260927 16:33:51.931771 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:51.957433 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:51.960904 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:51.961062 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:51.961611 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:51.962456 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:51.962507 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.962546 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:51.962563 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:51.971230 6176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35695
I20260927 16:33:51.971312 6304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35695 every 8 connection(s)
I20260927 16:33:51.972251 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:51.977870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6176
I20260927 16:33:51.977996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:51.978292 392 external_mini_cluster.cc:1428] Setting key c5261befa5273026404e1fc091c55dba
Sep 27 16:33:51 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:51.987285 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:51.974210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51655 (local address 127.0.98.62:39471)
0927 16:33:51.974339 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:51.974343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:51.975228 (+ 885us) server_negotiation.cc:407] Connection header received
0927 16:33:51.976641 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:51.976645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:51.976702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:51.976792 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:51.978018 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.978726 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.980228 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.980440 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.980523 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:51.983187 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:51.983193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:51.983200 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:51.983233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:51.985085 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.985596 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.985603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.985606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.985708 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:51.986027 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:51.986031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:51.986033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:51.986325 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:51.986426 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:51.986926 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 16:33:51.987067 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:33:51.988011 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:51.974614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39471 (local address 127.0.98.2:51655)
0927 16:33:51.975066 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:51.975111 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:51.976416 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:51.976995 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:51.977004 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:51.977371 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:51.977867 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:51.977881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.978889 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.978893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:51.980096 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:51.980109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:51.980597 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:51.980604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:51.980831 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:51.981479 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:51.981506 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:51.983009 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:51.985225 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:51.985230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:51.985242 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:51.985494 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:51.985805 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:51.985807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:51.985809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:51.985934 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:51.986436 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:51.986441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:51.986682 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 16:33:51.986838 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:33:51.987093 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:33:51.989203 6305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39471
I20260927 16:33:51.989420 6305 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:51.989912 6305 heartbeater.cc:511] Master 127.0.98.62:39471 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:33:51.991106 5980 ts_manager.cc:194] Registered new tserver with Master: d4622c57e86d4b5e954a3b8d403c44bb (127.0.98.2:35695)
I20260927 16:33:51.991950 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:33:52.005293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:52.115104 6310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:52.115463 6310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:52.115547 6310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:52.119897 6310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:52.119957 6310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:52.120018 6310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:52.125305 6310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:52.126405 6310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:52.127408 6310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:52.137296 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:52.137323 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:52.137296 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:52.137713 6310 server_base.cc:1034] running on GCE node
I20260927 16:33:52.138140 6310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:52.138868 6310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:52.140031 6310 hybrid_clock.cc:648] HybridClock initialized: now 1790526832140018 us; error 24 us; skew 500 ppm
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.142788 6310 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:33:52.143666 6310 webserver.cc:460] Webserver started at http://127.0.98.3:46513/ using document root <none> and password file <none>
I20260927 16:33:52.144191 6310 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:52.144248 6310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:52.144409 6310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:52.146427 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "5bf2a36837bc41f3bb09de0119e183d2"
format_stamp: "Formatted at 2026-09-27 16:33:52 on dist-test-slave-t27c"
server_key: "47f93aabfb34b032a9b03bf9ec5b2c96"
server_key_iv: "b2eee8921132897c4269c837b2a25795"
server_key_version: "encryptionkey@0"
I20260927 16:33:52.146924 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5bf2a36837bc41f3bb09de0119e183d2"
format_stamp: "Formatted at 2026-09-27 16:33:52 on dist-test-slave-t27c"
server_key: "47f93aabfb34b032a9b03bf9ec5b2c96"
server_key_iv: "b2eee8921132897c4269c837b2a25795"
server_key_version: "encryptionkey@0"
I20260927 16:33:52.150540 6310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:52.152889 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:52.154536 6310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:52.154673 6310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "5bf2a36837bc41f3bb09de0119e183d2"
format_stamp: "Formatted at 2026-09-27 16:33:52 on dist-test-slave-t27c"
server_key: "47f93aabfb34b032a9b03bf9ec5b2c96"
server_key_iv: "b2eee8921132897c4269c837b2a25795"
server_key_version: "encryptionkey@0"
I20260927 16:33:52.154785 6310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:52.171234 6310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:52.174259 6310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:52.174469 6310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:52.175109 6310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:52.176290 6310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:52.176358 6310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:52.176420 6310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:52.176460 6310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:52.186890 6310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32969
I20260927 16:33:52.186950 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32969 every 8 connection(s)
I20260927 16:33:52.187812 6310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:52.191457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6310
I20260927 16:33:52.191572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:52.191870 392 external_mini_cluster.cc:1428] Setting key 6dd31081d11e9a18839a11d3c67106bc
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:33:52.201731 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:52.189587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32831 (local address 127.0.98.62:39471)
0927 16:33:52.189739 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.189743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.190576 (+ 833us) server_negotiation.cc:407] Connection header received
0927 16:33:52.192298 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.192301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.192358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.192466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.193939 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.194526 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.195712 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.195927 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.196038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:52.198385 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:52.198389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:52.198391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:52.198413 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:52.199632 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.200128 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.200132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.200134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.200306 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.200684 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.200689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.200691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.200866 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:52.200971 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:52.201449 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 16:33:52.201570 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:33:52.202605 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:52.189969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39471 (local address 127.0.98.3:32831)
0927 16:33:52.190428 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:52.190471 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:52.192049 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:52.192676 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:52.192684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:52.193202 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:52.193797 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:52.193817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.194649 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.194653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:52.195591 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:52.195599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.196073 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.196077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:52.196257 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:52.196784 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:52.196800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:52.198217 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:52.199748 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:52.199754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:52.199768 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:52.200023 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:52.200428 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:52.200432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:52.200434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:52.200575 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:52.200979 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:52.200987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:52.201221 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 16:33:52.201411 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 16:33:52.201742 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
I20260927 16:33:52.203876 6439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39471
I20260927 16:33:52.204082 6439 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:52.204631 6439 heartbeater.cc:511] Master 127.0.98.62:39471 requested a full tablet report, sending...
I20260927 16:33:52.205608 5979 ts_manager.cc:194] Registered new tserver with Master: 5bf2a36837bc41f3bb09de0119e183d2 (127.0.98.3:32969)
I20260927 16:33:52.206262 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32831
I20260927 16:33:52.217715 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/test-admin.keytab.
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.234014 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 16:33:52.237195 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:52.235681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.98.1:43369)
0927 16:33:52.235944 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.235962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.235992 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:33:52.236042 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.236047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.236261 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.236430 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.236973 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41600: BlockingRecv error: recv got EOF from 127.0.0.1:41600 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
W20260927 16:33:52.237269 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.238382 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.239275 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:52.248327 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:52.240083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.98.1:43369)
0927 16:33:52.240225 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.240228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.240242 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:33:52.240338 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.240340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.240376 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.240443 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.240918 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.241685 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.242588 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.242970 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.243058 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:52.245146 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:52.245155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:52.245168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:52.245190 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:52.246815 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.247184 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.247189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.247190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.247261 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.247473 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.247475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.247476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.247691 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:52.247836 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:52.247978 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 16:33:52.248156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 16:33:52.252085 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:52.251236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.0.98.1:43369)
0927 16:33:52.251372 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.251375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.251388 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:52.251494 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.251496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.251530 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.251594 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.251971 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41620: BlockingRecv error: recv got EOF from 127.0.0.1:41620 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 16:33:52.252143 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.253146 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.253991 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:52.261822 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:52.254855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.98.1:43369)
0927 16:33:52.254983 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.254992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.255005 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:52.255105 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.255107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.255143 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.255208 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.255643 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.256151 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.256868 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.257078 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.257153 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:52.259223 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:52.259227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:52.259229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:52.259249 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:52.260466 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.260832 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.260834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.260835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.260903 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.261118 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.261121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.261122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.261263 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:52.261356 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:52.261521 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 16:33:52.261604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 16:33:52.264290 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:52.263508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.1:43369)
0927 16:33:52.263638 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.263646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.263658 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:33:52.263757 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.263759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.263790 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.263851 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.264177 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41644: BlockingRecv error: recv got EOF from 127.0.0.1:41644 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 16:33:52.264359 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526802550915-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.265257 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:52.266078 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:33:52.274766 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:52.267249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.98.1:43369)
0927 16:33:52.267384 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:52.267388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:52.267464 (+ 76us) server_negotiation.cc:407] Connection header received
0927 16:33:52.267574 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:52.267577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:52.267617 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:52.267707 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:52.268272 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.268746 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.269520 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:52.269747 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:52.269847 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:52.272109 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:52.272112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:52.272114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:52.272135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:52.273503 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.273849 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.273852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.273853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.273916 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:52.274118 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:52.274120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:52.274121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:52.274315 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:52.274407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:52.274573 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:33:52.274659 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:33:52.275703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6042
I20260927 16:33:52.281157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6176
I20260927 16:33:52.286352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6310
I20260927 16:33:52.291713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 5948
2026-09-27T16:33:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2165 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[6459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[6459](info): set up 2 sockets
Sep 27 16:33:52 dist-test-slave-t27c krb5kdc[6459](info): commencing operation
krb5kdc: starting...
W20260927 16:33:53.330678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:53 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 16:33:53.449317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:53.559890 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:53.560204 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:53.560303 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:53.564508 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:53.564585 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:53.564622 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:53.564677 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:53.564714 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:53.570017 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:53.571250 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:53.572425 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:53.581319 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:53.581498 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:53.581513 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:53.581563 6472 server_base.cc:1034] running on GCE node
I20260927 16:33:53.582310 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:53.583478 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:53.584638 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790526833584610 us; error 39 us; skew 500 ppm
Sep 27 16:33:53 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:53.587510 6472 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 16:33:53.588364 6472 webserver.cc:460] Webserver started at http://127.0.98.62:45803/ using document root <none> and password file <none>
I20260927 16:33:53.588910 6472 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:53.588976 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:53.589227 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:53.591359 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "985d4c004a2f4febae194330d09193bb"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "34217791a6c92c4a5c5a16d12edd15cd"
server_key_iv: "f05776bb091f7eadc623db0788bce6b9"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.591917 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "985d4c004a2f4febae194330d09193bb"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "34217791a6c92c4a5c5a16d12edd15cd"
server_key_iv: "f05776bb091f7eadc623db0788bce6b9"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.595530 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:53.597787 6487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:53.599066 6472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:53.599216 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "985d4c004a2f4febae194330d09193bb"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "34217791a6c92c4a5c5a16d12edd15cd"
server_key_iv: "f05776bb091f7eadc623db0788bce6b9"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.599345 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:53.634807 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:53.637759 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:53.637918 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:53.644810 6472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36969
I20260927 16:33:53.645123 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36969 every 8 connection(s)
I20260927 16:33:53.645830 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:53.648586 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:53.654078 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Bootstrap starting.
I20260927 16:33:53.655355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6472
I20260927 16:33:53.655498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:53.655790 392 external_mini_cluster.cc:1428] Setting key 1e0b5dbb8ce3066076703cfb04f73fe7
I20260927 16:33:53.657279 6540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:53.658022 6540 log.cc:824] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:53.659893 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: No bootstrap required, opened a new log
I20260927 16:33:53.662853 6540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260927 16:33:53.663092 6540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:53.663170 6540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985d4c004a2f4febae194330d09193bb, State: Initialized, Role: FOLLOWER
I20260927 16:33:53.663882 6540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260927 16:33:53.664034 6540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:53.664119 6540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:53.664238 6540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:33:53 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:53.665098 6540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260927 16:33:53.665452 6540 leader_election.cc:302] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 985d4c004a2f4febae194330d09193bb; no voters:
I20260927 16:33:53.665757 6540 leader_election.cc:288] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:53.665874 6545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:53.666113 6545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 LEADER]: Becoming Leader. State: Replica: 985d4c004a2f4febae194330d09193bb, State: Running, Role: LEADER
I20260927 16:33:53.666527 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260927 16:33:53.667459 6540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:53.668378 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260927 16:33:53.668512 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:53.668732 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985d4c004a2f4febae194330d09193bb. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260927 16:33:53.668867 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:53.671885 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:53.656956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.98.62:36969)
0927 16:33:53.657406 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:53.657429 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:53.657477 (+ 48us) server_negotiation.cc:407] Connection header received
0927 16:33:53.658175 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:53.658232 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:53.658588 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:53.658934 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:53.659751 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.660780 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.661646 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.662712 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.662905 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:53.665787 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:53.665794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:53.665820 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:53.665854 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:53.668153 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:53.668685 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:53.668693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:53.668702 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:53.668881 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:53.669277 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:53.669280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:53.669282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:53.669846 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:53.670046 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:53.670786 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 16:33:53.671109 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
W20260927 16:33:53.672696 6560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:53.672775 6560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:53.672839 6554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:53.673689 6554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:53.679304 6554 catalog_manager.cc:1347] Generated new cluster ID: 8c0f8f483f42473a8a0d38aff9c0e13b
I20260927 16:33:53.679386 6554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:53.691409 6554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:53.692389 6554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:53.701355 6554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Generated new TSK 0
I20260927 16:33:53.702096 6554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 16:33:53.731585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:53.847525 6566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:53.847780 6566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:53.847826 6566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:53.851892 6566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:53.851962 6566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:53.852068 6566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:53.857486 6566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:53.858685 6566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:53.859774 6566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:53.869387 6572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:53.869429 6571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:53.869557 6566 server_base.cc:1034] running on GCE node
W20260927 16:33:53.869588 6574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:53.870127 6566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:53.870890 6566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:53.872087 6566 hybrid_clock.cc:648] HybridClock initialized: now 1790526833872075 us; error 34 us; skew 500 ppm
Sep 27 16:33:53 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:53.874835 6566 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:33:53.875651 6566 webserver.cc:460] Webserver started at http://127.0.98.1:41789/ using document root <none> and password file <none>
I20260927 16:33:53.876174 6566 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:53.876231 6566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:53.876394 6566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:53.878423 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ea7b9eb8f144161b922ae78202486be"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "11e02c552ed088f30e92416aae10b0c2"
server_key_iv: "6848bfc0086078ec9943589dc6df8ab4"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.878939 6566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea7b9eb8f144161b922ae78202486be"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "11e02c552ed088f30e92416aae10b0c2"
server_key_iv: "6848bfc0086078ec9943589dc6df8ab4"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.882450 6566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:53.884732 6581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:53.885913 6566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:53.886005 6566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "5ea7b9eb8f144161b922ae78202486be"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "11e02c552ed088f30e92416aae10b0c2"
server_key_iv: "6848bfc0086078ec9943589dc6df8ab4"
server_key_version: "encryptionkey@0"
I20260927 16:33:53.886085 6566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:53.903270 6566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:53.906391 6566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:53.906554 6566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:53.907094 6566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:53.907882 6566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:53.907927 6566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:53.907966 6566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:53.907991 6566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:53.916836 6566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39417
I20260927 16:33:53.916860 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39417 every 8 connection(s)
I20260927 16:33:53.917733 6566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:53.918560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6566
I20260927 16:33:53.918676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:53.918949 392 external_mini_cluster.cc:1428] Setting key 3bca067f04faa2d924b86b40843a9ae8
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 16:33:53 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 16:33:53.933349 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:53.919608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41599 (local address 127.0.98.62:36969)
0927 16:33:53.919742 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:53.919746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:53.920708 (+ 962us) server_negotiation.cc:407] Connection header received
0927 16:33:53.922277 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:53.922279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:53.922324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:53.922385 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:53.924009 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.924618 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.926472 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.926706 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.926782 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:53.929608 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:53.929612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:53.929615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:53.929642 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:53.931064 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:53.931686 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:53.931690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:53.931692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:53.931785 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:53.932109 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:53.932115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:53.932120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:53.932376 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:53.932480 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:53.933023 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 16:33:53.933166 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:53.934357 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:53.920027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:41599)
0927 16:33:53.920542 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:53.920596 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:53.922018 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:53.922579 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:53.922588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:53.923148 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:53.923856 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:53.923878 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.924754 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.924759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:53.926358 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:53.926372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:53.926870 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:53.926875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:53.927094 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:53.927699 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:53.927718 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:53.929435 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:53.931189 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:53.931197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:53.931215 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:53.931584 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:53.931877 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:53.931879 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:53.931881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:53.932016 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:53.932485 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:53.932490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:53.932734 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 16:33:53.932943 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 16:33:53.933300 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
I20260927 16:33:53.935877 6695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:53.936141 6695 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:53.936658 6695 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:53.938267 6504 ts_manager.cc:194] Registered new tserver with Master: 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417)
I20260927 16:33:53.939407 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 16:33:53.947211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:54.059994 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:54.060341 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:54.060487 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:54.064826 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:54.064927 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:54.065028 6700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:54.070940 6700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:54.072126 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:54.073143 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:54.082724 6706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.082711 6705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.082724 6708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.083099 6700 server_base.cc:1034] running on GCE node
I20260927 16:33:54.083447 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:54.084086 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:54.085239 6700 hybrid_clock.cc:648] HybridClock initialized: now 1790526834085224 us; error 27 us; skew 500 ppm
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:54.087956 6700 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:33:54.088864 6700 webserver.cc:460] Webserver started at http://127.0.98.2:38539/ using document root <none> and password file <none>
I20260927 16:33:54.089378 6700 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.089432 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.089617 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:54.091709 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b47527560b24b25974b6915e16d69ba"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "f6532c3a5c77bb396298b4638ad34c2f"
server_key_iv: "3903b1844c1213e44115c6daf033c6eb"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.092298 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b47527560b24b25974b6915e16d69ba"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "f6532c3a5c77bb396298b4638ad34c2f"
server_key_iv: "3903b1844c1213e44115c6daf033c6eb"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.096233 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 16:33:54.098688 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.099995 6700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:54.100126 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "6b47527560b24b25974b6915e16d69ba"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "f6532c3a5c77bb396298b4638ad34c2f"
server_key_iv: "3903b1844c1213e44115c6daf033c6eb"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.100238 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.120477 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:54.124270 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:54.124485 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:54.125087 6700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:54.125887 6700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:54.125955 6700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.126024 6700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:54.126075 6700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.134548 6700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41765
I20260927 16:33:54.134568 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41765 every 8 connection(s)
I20260927 16:33:54.135504 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:54.143569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6700
I20260927 16:33:54.143715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:54.143994 392 external_mini_cluster.cc:1428] Setting key dc790610765d911348b29e49a0f96605
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:54.150084 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.137413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60363 (local address 127.0.98.62:36969)
0927 16:33:54.137558 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.137562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.138486 (+ 924us) server_negotiation.cc:407] Connection header received
0927 16:33:54.139839 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.139842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.139899 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.140024 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.141184 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.141706 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.142939 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.143163 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.143279 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.146081 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.146087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.146091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.146127 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.147790 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.148426 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.148432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.148438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.148530 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.148840 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.148847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.148851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.149124 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.149229 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.149687 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.149832 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:33:54.150806 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.137819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:60363)
0927 16:33:54.138333 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.138378 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.139628 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.140193 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.140200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.140566 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.141039 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.141053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.141831 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.141836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.142808 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.142817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.143339 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.143345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.143521 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.144152 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.144167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.145904 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.147904 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.147911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.147928 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.148303 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.148620 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.148622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.148623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.148747 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.149223 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.149228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.149445 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.149598 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.149915 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260927 16:33:54.152011 6829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:54.152271 6829 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:54.152855 6829 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:54.154011 6504 ts_manager.cc:194] Registered new tserver with Master: 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:33:54.154624 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:33:54.172641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:54.288553 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:54.288851 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:54.288926 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:54.293028 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:54.293128 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:54.293218 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:54.298605 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:54.299741 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:54.300807 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:54.310638 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.310624 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.310684 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.310828 6834 server_base.cc:1034] running on GCE node
I20260927 16:33:54.311384 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:54.312029 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:54.313212 6834 hybrid_clock.cc:648] HybridClock initialized: now 1790526834313189 us; error 39 us; skew 500 ppm
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:54.315975 6834 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:33:54.316812 6834 webserver.cc:460] Webserver started at http://127.0.98.3:34619/ using document root <none> and password file <none>
I20260927 16:33:54.317379 6834 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.317484 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.317708 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:54.319774 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "77c631d37446474a9fd23eac9594857a"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "96dc3614e32913ae9c33181be927c90f"
server_key_iv: "6875950aa7f1fad70b62d66d1d50cf6d"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.320345 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77c631d37446474a9fd23eac9594857a"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "96dc3614e32913ae9c33181be927c90f"
server_key_iv: "6875950aa7f1fad70b62d66d1d50cf6d"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.323971 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:33:54.326289 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.327513 6834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:54.327643 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "77c631d37446474a9fd23eac9594857a"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "96dc3614e32913ae9c33181be927c90f"
server_key_iv: "6875950aa7f1fad70b62d66d1d50cf6d"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.327752 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.344745 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:54.348264 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:54.348486 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:54.349092 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:54.349915 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:54.349988 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.350057 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:54.350112 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.358620 6834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36955
I20260927 16:33:54.358646 6962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36955 every 8 connection(s)
I20260927 16:33:54.359549 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:54.369156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6834
I20260927 16:33:54.369287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:54.369616 392 external_mini_cluster.cc:1428] Setting key bcf61c3ec9033984b6193231c30de325
I20260927 16:33:54.373514 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.361253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35933 (local address 127.0.98.62:36969)
0927 16:33:54.361387 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.361394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.362159 (+ 765us) server_negotiation.cc:407] Connection header received
0927 16:33:54.363525 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.363529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.363589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.363669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.364813 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.365352 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.366537 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.366761 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.366874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.369632 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.369636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.369638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.369680 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.371526 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.372054 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.372057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.372058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.372136 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.372543 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.372545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.372546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.372734 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.372815 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.373286 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.373389 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:54.374502 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.361560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:35933)
0927 16:33:54.362002 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.362045 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.363307 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.363834 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.363841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.364185 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.364661 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.364674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.365487 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.365493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.366396 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.366405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.366915 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.366918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.367093 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.367697 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.367712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.369406 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.371633 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.371637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.371670 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.371973 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.372218 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.372220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.372221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.372440 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.372825 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.372829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.373076 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.373235 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.373512 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260927 16:33:54.375643 6963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:54.375898 6963 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:54.376407 6963 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:54.377398 6504 ts_manager.cc:194] Registered new tserver with Master: 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:54.377976 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35933
I20260927 16:33:54.383644 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 16:33:54.408962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44811
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36969,127.0.98.61:44811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:54.520061 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:54.520318 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:54.520356 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:54.524379 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:54.524432 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:54.524451 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:54.524503 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:54.524535 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:54.529728 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36969,127.0.98.61:44811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
W20260927 16:33:54.529963 6968 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36969,127.0.98.61:44811'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 16:33:54.530846 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:54.531952 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:54.540911 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.540905 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.541019 6968 server_base.cc:1034] running on GCE node
W20260927 16:33:54.540910 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.541778 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:54.542984 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:54.544140 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790526834544131 us; error 45 us; skew 500 ppm
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:54.546831 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 16:33:54.547739 6968 webserver.cc:460] Webserver started at http://127.0.98.61:40275/ using document root <none> and password file <none>
I20260927 16:33:54.548297 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.548383 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.548619 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:54.551275 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/instance:
uuid: "73b0bbc7de2342e589dab032e20bea59"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "995bf9c6d1ac82b2e3ab48fda1fc5149"
server_key_iv: "6e1643058ead96d84f8adda14521d9d1"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.552034 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal/instance:
uuid: "73b0bbc7de2342e589dab032e20bea59"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "995bf9c6d1ac82b2e3ab48fda1fc5149"
server_key_iv: "6e1643058ead96d84f8adda14521d9d1"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.556973 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 16:33:54.559538 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.560902 6968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:54.561043 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "73b0bbc7de2342e589dab032e20bea59"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "995bf9c6d1ac82b2e3ab48fda1fc5149"
server_key_iv: "6e1643058ead96d84f8adda14521d9d1"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.561156 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.572803 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:54.575984 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:54.576201 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:54.583045 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44811
I20260927 16:33:54.583310 7035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44811 every 8 connection(s)
I20260927 16:33:54.584034 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:33:54.585119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 6968
I20260927 16:33:54.587108 7036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:54.592458 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:33:54.599983 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.586609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.61:44811)
0927 16:33:54.587041 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.587056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.587091 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:33:54.587773 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.587790 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.588126 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.588463 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.589287 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.590568 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.591434 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.591903 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.592097 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.595121 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.595127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.595143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.595172 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.596992 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.597444 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.597450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.597456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.597573 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.597849 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.597854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.597856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.598375 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.598582 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.598826 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.599114 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:54.603777 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.594676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.98.62:36969)
0927 16:33:54.594797 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.594802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.595399 (+ 597us) server_negotiation.cc:407] Connection header received
0927 16:33:54.595933 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.595936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.595992 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.596098 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.596798 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.597322 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.598246 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.598463 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.598577 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.600790 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.600794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.600796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.600825 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.602317 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.602701 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.602703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.602704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.602784 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.603038 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.603040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.603041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.603229 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.603304 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.603579 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.603667 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:33:54.603785 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.594633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43314)
0927 16:33:54.595265 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.595316 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.595797 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.596126 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.596131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.596528 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.596635 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.596641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.597436 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.597442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.598086 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.598094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.598577 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.598581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.598712 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.599254 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.599265 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.600656 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.602403 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.602409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.602413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.602617 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.602859 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.602861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.602862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.602950 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.603313 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.603318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.603409 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.603523 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.603652 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":159,"threads_started":1}
I20260927 16:33:54.605032 7036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:33:54.612581 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.605562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56652)
0927 16:33:54.605682 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.605697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.605796 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.606030 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.606032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.606154 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.606322 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.606327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.607381 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.607383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.607872 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.607878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.608258 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.608261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.608332 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.608696 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.608706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.609795 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.611421 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.611423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.611425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.611602 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.611821 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.611823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.611824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.611890 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.612184 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.612186 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.612253 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.612330 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.612444 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 16:33:54.612607 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.605592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.0.98.61:44811)
0927 16:33:54.605682 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.605687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.605786 (+ 99us) server_negotiation.cc:407] Connection header received
0927 16:33:54.605889 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.605892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.605929 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.606005 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.606411 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.607295 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.607959 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.608174 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.608245 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.609913 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.609916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.609919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.609941 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.611331 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.611681 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.611683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.611684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.611749 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.611959 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.611961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.611962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.612110 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.612181 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.612372 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.612488 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260927 16:33:54.615717 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Bootstrap starting.
I20260927 16:33:54.618057 7036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:54.618888 7036 log.cc:824] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:54.620712 7036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: No bootstrap required, opened a new log
I20260927 16:33:54.623726 7036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } }
I20260927 16:33:54.623917 7036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:54.623997 7036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b0bbc7de2342e589dab032e20bea59, State: Initialized, Role: FOLLOWER
I20260927 16:33:54.624404 7036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } }
I20260927 16:33:54.624856 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } } }
I20260927 16:33:54.625005 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:54.625569 7036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:33:54.630065 7053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:54.630153 7053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:54.631953 6968 master_runner.cc:428] Detected that this master 73b0bbc7de2342e589dab032e20bea59 is joining an existing cluster
I20260927 16:33:54.632025 6968 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44811
I20260927 16:33:54.642092 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.634497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.61:44811)
0927 16:33:54.634595 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.634599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.634614 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:33:54.635113 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.635116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.635156 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.635246 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.635704 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.636183 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.637080 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.637288 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.637395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.638666 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.638672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.638675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.638702 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.640446 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.640902 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.640907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.640910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.640995 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641248 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.641255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.641257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.641427 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.641528 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.641792 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.641894 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260927 16:33:54.642571 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.634129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56654)
0927 16:33:54.634525 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.634540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.634645 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.635273 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.635276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.635485 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.635592 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.635597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.636294 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.636298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.636972 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.636980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.637401 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.637404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.637498 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.638029 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.638045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.638523 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.640567 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.640570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.640574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.640800 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.641093 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.641097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.641178 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.641528 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.641531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.641616 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.642258 (+ 642us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.642383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":144,"threads_started":1}
I20260927 16:33:54.642906 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.634037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43330)
0927 16:33:54.634496 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.634511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.634647 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.635019 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.635021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.635192 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.635354 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.635363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.636609 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.636612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.637583 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.637594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.638233 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.638236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.638323 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.638788 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.638802 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.639194 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.641204 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.641213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.641459 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.641773 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.641780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.641865 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.642339 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.642344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.642439 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.642576 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.642724 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":30,"spinlock_wait_cycles":6784,"thread_start_us":93,"threads_started":1}
I20260927 16:33:54.643198 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.634094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.98.62:36969)
0927 16:33:54.634243 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.634248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.634585 (+ 337us) server_negotiation.cc:407] Connection header received
0927 16:33:54.634836 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.634840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.634899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.634996 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.635493 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.636441 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.637756 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.638093 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.638257 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.639339 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.639342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.639345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.639375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.641101 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641589 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.641593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.641594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.641678 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.641967 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.641970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.641972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.642204 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.642297 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.642926 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.643048 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:33:54.660641 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.647530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56666)
0927 16:33:54.647907 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.647956 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.648068 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.648645 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.648651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.649075 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.649194 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.649201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.650200 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.650203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.651175 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.651185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.651911 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.651914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.652024 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.652653 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.652667 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.656352 (+ 3685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.658449 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.658460 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.658467 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.658796 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.659124 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.659127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.659129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.659225 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.659755 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.659766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.659940 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.660110 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.660367 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260927 16:33:54.661491 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.648261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.0.98.61:44811)
0927 16:33:54.648382 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.648386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.648399 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:54.648457 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.648460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.648508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.648596 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.649316 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.650061 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.651455 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.651791 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.651897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.656545 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.656549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.656553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.656580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.658325 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.658912 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.658916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.658918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.659007 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.659327 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.659330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.659332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.659586 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.659690 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.661105 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.661253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:54.663353 6504 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44811: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "73b0bbc7de2342e589dab032e20bea59"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44811
}
attrs {
promote: true
}
}
dest_uuid: "985d4c004a2f4febae194330d09193bb"
cas_config_opid_index: -1
I20260927 16:33:54.664192 6504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 16:33:54.666283 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } }
I20260927 16:33:54.667014 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 16:33:54.668320 7064 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44811
I20260927 16:33:54.668550 7010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 16:33:54.669056 6968 master.cc:537] Master@127.0.98.61:44811 shutting down...
I20260927 16:33:54.669585 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:54.669929 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } }
I20260927 16:33:54.670292 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
W20260927 16:33:54.672309 6489 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 16:33:54.673203 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb -> Peer 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61:44811): Couldn't send request to peer 73b0bbc7de2342e589dab032e20bea59. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 16:33:54.673751 6968 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:54.674031 6968 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:54.674160 6968 tablet_replica.cc:330] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 16:33:54.688868 6968 master.cc:552] Master@127.0.98.61:44811 shutdown complete.
I20260927 16:33:54.688941 6968 master_runner.cc:305] Clearing existing system tablet
I20260927 16:33:54.689234 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.689292 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.689276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33037
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:33:54.690871 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 16:33:54.691836 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.692083 6968 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.692212 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:54.692246 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "73b0bbc7de2342e589dab032e20bea59"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "995bf9c6d1ac82b2e3ab48fda1fc5149"
server_key_iv: "6e1643058ead96d84f8adda14521d9d1"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.694744 6968 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:54.697310 6968 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:33:54.697429 6968 log.cc:1196] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 16:33:54.697731 6968 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Deleting consensus metadata
I20260927 16:33:54.697898 6968 master_runner.cc:315] Copying system tablet from 127.0.98.62:36969
I20260927 16:33:54.699167 6968 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36969
I20260927 16:33:54.708791 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.699990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.0.98.62:36969)
0927 16:33:54.700124 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.700128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.700389 (+ 261us) server_negotiation.cc:407] Connection header received
0927 16:33:54.700531 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.700534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.700588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.700690 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.701220 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.702051 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.703391 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.703755 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.703833 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.705290 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.705294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.705297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.705324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.707083 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.707562 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.707565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.707567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.707650 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.707924 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.707927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.707929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.708137 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.708234 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.708511 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.708627 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:54.708802 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.699899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43342)
0927 16:33:54.700270 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.700287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.700420 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.700720 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.700723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.700962 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.701092 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.701099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.702211 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.702215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.703214 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.703225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.703992 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.703995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.704090 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.704674 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.704688 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.705118 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.707200 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.707204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.707206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.707463 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.707742 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.707744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.707746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.707834 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.708232 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.708235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.708330 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.708447 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.708595 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260927 16:33:54.709717 6524 tablet_copy_service.cc:140] P 985d4c004a2f4febae194330d09193bb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 73b0bbc7de2342e589dab032e20bea59 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43342)
I20260927 16:33:54.709905 6524 tablet_copy_service.cc:161] P 985d4c004a2f4febae194330d09193bb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 73b0bbc7de2342e589dab032e20bea59 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43342: session id = 73b0bbc7de2342e589dab032e20bea59-00000000000000000000000000000000
I20260927 16:33:54.711817 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:33:54.713563 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:33:54.715919 6968 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: tablet copy: Starting download of 0 data blocks...
I20260927 16:33:54.716132 6968 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: tablet copy: Starting download of 1 WAL segments...
I20260927 16:33:54.717885 6968 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:33:54.719771 6524 tablet_copy_service.cc:342] P 985d4c004a2f4febae194330d09193bb: Request end of tablet copy session 73b0bbc7de2342e589dab032e20bea59-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43342
I20260927 16:33:54.719895 6524 tablet_copy_service.cc:433] P 985d4c004a2f4febae194330d09193bb: ending tablet copy session 73b0bbc7de2342e589dab032e20bea59-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 73b0bbc7de2342e589dab032e20bea59
I20260927 16:33:54.727412 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:54.729327 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.731181 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.731380 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.737373 6968 server_base.cc:1034] running on GCE node
I20260927 16:33:54.737591 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:54.737920 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:54.739070 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790526834739056 us; error 35 us; skew 500 ppm
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:54.740774 6968 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 16:33:54.741274 6968 webserver.cc:460] Webserver started at http://127.0.98.61:35073/ using document root <none> and password file <none>
I20260927 16:33:54.741480 6968 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.741523 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.743096 6968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:54.744263 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.744498 6968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:54.744561 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "73b0bbc7de2342e589dab032e20bea59"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "995bf9c6d1ac82b2e3ab48fda1fc5149"
server_key_iv: "6e1643058ead96d84f8adda14521d9d1"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.744645 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.771437 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:54.773819 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:54.778805 6968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44811
I20260927 16:33:54.778865 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44811 every 8 connection(s)
I20260927 16:33:54.779524 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:33:54.781316 7141 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:33:54.782474 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Bootstrap starting.
I20260927 16:33:54.792675 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:33:54.792984 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: Bootstrap complete.
I20260927 16:33:54.793573 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } }
I20260927 16:33:54.793721 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73b0bbc7de2342e589dab032e20bea59, State: Initialized, Role: LEARNER
I20260927 16:33:54.793819 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } }
I20260927 16:33:54.794229 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } }
I20260927 16:33:54.794360 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:54.794430 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:33:54.797384 7155 catalog_manager.cc:1259] Loaded cluster ID: 8c0f8f483f42473a8a0d38aff9c0e13b
I20260927 16:33:54.797513 7155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: loading cluster ID for follower catalog manager: success
I20260927 16:33:54.799611 7155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: acquiring CA information for follower catalog manager: success
I20260927 16:33:54.800932 7155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:33:54.826122 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:54.826440 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:54.826480 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:54.830485 7069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:54.830546 7069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:54.830566 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:54.830582 7069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:54.830597 7069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:54.836040 7069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:54.837108 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:54.838133 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:54.846411 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.846495 7069 server_base.cc:1034] running on GCE node
W20260927 16:33:54.846418 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:54.846792 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:54.847405 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:54.848498 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:54.849680 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790526834849653 us; error 48 us; skew 500 ppm
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:54.852439 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 16:33:54.853343 7069 webserver.cc:460] Webserver started at http://127.0.98.60:42013/ using document root <none> and password file <none>
I20260927 16:33:54.853943 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:54.854039 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:54.854337 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:54.856546 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/instance:
uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "bd082ce4484dbc7aa2ede4438892fd44"
server_key_iv: "1d1864ad8c8671084d526bd68626fee5"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.857142 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal/instance:
uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "bd082ce4484dbc7aa2ede4438892fd44"
server_key_iv: "1d1864ad8c8671084d526bd68626fee5"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.860920 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260927 16:33:54.863420 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:54.864672 7069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:54.864812 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "bd082ce4484dbc7aa2ede4438892fd44"
server_key_iv: "1d1864ad8c8671084d526bd68626fee5"
server_key_version: "encryptionkey@0"
I20260927 16:33:54.864928 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:54.883529 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:54.888126 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:54.888336 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:54.896896 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33037
I20260927 16:33:54.896944 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33037 every 8 connection(s)
I20260927 16:33:54.897879 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:33:54.900717 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:54.904696 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:33:54.906662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7069
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:33:54.918424 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.906661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.98.62:36969)
0927 16:33:54.906790 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.906800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.908060 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 16:33:54.908840 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.908844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.908905 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.908994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.909771 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.910360 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.911546 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.911759 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.911849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.914631 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.914640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.914643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.914666 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.916190 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.916794 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.916799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.916801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.916897 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.917269 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.917272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.917274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.917538 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.917632 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.918109 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.918254 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:33:54.919373 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.907103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43358)
0927 16:33:54.907882 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.907949 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.908650 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.909041 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.909046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.909391 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.909629 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.909641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.910485 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.910488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.911428 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.911437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.911926 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.911932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.912087 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.912713 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.912737 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.914480 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.916302 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.916311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.916321 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.916685 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.917004 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.917006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.917008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.917164 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.917627 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.917633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.917880 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.918070 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.918406 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":123,"threads_started":1}
I20260927 16:33:54.919379 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.907468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.0.98.60:33037)
0927 16:33:54.907974 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.907983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.908015 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:33:54.908635 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.908651 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.908840 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.909028 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.909823 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.910912 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.911950 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.912326 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.912439 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.915045 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.915053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.915067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.915095 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.916921 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.917444 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.917450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.917451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.917552 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.917862 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.917865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.917867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.918113 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.918948 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.919047 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.919179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":117,"threads_started":1}
I20260927 16:33:54.920666 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:33:54.928395 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.921192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56674)
0927 16:33:54.921295 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.921309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.921385 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.921783 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.921785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.921912 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.922034 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.922041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.923023 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.923026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.923582 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.923589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.923985 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.923988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.924057 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.924441 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.924451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.925535 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.927257 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.927259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.927260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.927434 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.927658 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.927660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.927661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.927727 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.928050 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.928052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.928120 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.928194 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.928284 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 16:33:54.928447 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.921283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.98.61:44811)
0927 16:33:54.921571 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.921575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.921589 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:33:54.921628 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.921631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.921674 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.921763 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.922151 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.922900 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.923678 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.923893 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.923971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.925656 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.925659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.925660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.925680 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.927170 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.927514 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.927516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.927517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.927584 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.927798 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.927800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.927801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.927977 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.928049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.928224 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.928311 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260927 16:33:54.929141 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:33:54.936131 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.929607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57432)
0927 16:33:54.929713 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.929728 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.929787 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.929967 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.929969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.930084 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.930146 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.930150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.930847 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.930849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.931327 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.931333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.931782 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.931785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.931840 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.932200 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.932210 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.933286 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.934920 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.934922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.934923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.935095 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.935321 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.935323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.935324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.935398 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.935699 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.935701 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.935770 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.935845 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.935960 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 16:33:54.936138 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.929641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.60:33037)
0927 16:33:54.929746 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.929749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.929774 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:33:54.929852 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.929853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.929884 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.929942 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.930283 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.930764 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.931455 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.931680 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.931754 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.933401 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.933404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.933406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.933448 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.934820 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.935179 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.935181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.935182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.935247 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.935472 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.935474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.935475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.935626 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.935695 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.935895 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.936013 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":40}
I20260927 16:33:54.939587 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Bootstrap starting.
I20260927 16:33:54.941897 6695 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
I20260927 16:33:54.941922 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:54.943252 7223 log.cc:824] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:54.945150 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: No bootstrap required, opened a new log
I20260927 16:33:54.948148 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } }
I20260927 16:33:54.948354 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:54.948434 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba3986e51f248c1b4c9dd5c4a724a9c, State: Initialized, Role: FOLLOWER
I20260927 16:33:54.948881 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } }
I20260927 16:33:54.949359 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } } }
I20260927 16:33:54.949494 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:54.950068 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:33:54.954480 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:33:54.954562 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:33:54.957782 7069 master_runner.cc:428] Detected that this master 1ba3986e51f248c1b4c9dd5c4a724a9c is joining an existing cluster
I20260927 16:33:54.957842 7069 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33037
I20260927 16:33:54.967890 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.98.61:44811)
0927 16:33:54.960331 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.960336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.960803 (+ 467us) server_negotiation.cc:407] Connection header received
0927 16:33:54.961005 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.961009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.961067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.961121 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.961816 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.962429 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.963251 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.963482 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.963598 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.964809 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.964813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.964817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.964845 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.966272 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.966696 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.966701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.966703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.966780 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.967090 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.967094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.967096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.967267 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.967375 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.967637 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.967749 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:33:54.967919 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56686)
0927 16:33:54.960719 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.960734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.960871 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.961264 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.961267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.961589 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.961702 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.961708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.962539 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.962547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.963139 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.963146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.963588 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.963592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.963710 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.964355 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.964364 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.964683 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.966383 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.966386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.966388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.966597 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.966868 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.966872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.966874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.966965 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.967366 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.967369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.967463 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.967590 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.967728 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260927 16:33:54.974787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6566
I20260927 16:33:54.981011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39417
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36969
--webserver_port=41789
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:33:54.987474 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.98.60:33037)
0927 16:33:54.960590 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.960595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.960900 (+ 305us) server_negotiation.cc:407] Connection header received
0927 16:33:54.961043 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.961046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.961080 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.961124 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.961855 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.962634 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.963808 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.964052 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.964156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.983547 (+ 19391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.983554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.983560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.983590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.985361 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.985880 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.985886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.985889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.985983 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.986342 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.986348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.986351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.986601 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.986680 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.987069 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.987242 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:33:54.988129 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57440)
0927 16:33:54.960821 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.960834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.960951 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.961250 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.961253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.961587 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.961741 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.961748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.962773 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.962776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.963695 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.963705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.964155 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.964158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.964261 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.982847 (+ 18586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.982868 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.983377 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.985490 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.985496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.985499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.985771 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.986090 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.986095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.986098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.986239 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.986749 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.986754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.986859 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.987699 (+ 840us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.987857 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":291,"thread_start_us":92,"threads_started":1}
I20260927 16:33:54.991825 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.0.98.62:36969)
0927 16:33:54.960462 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.960467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.960956 (+ 489us) server_negotiation.cc:407] Connection header received
0927 16:33:54.961106 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.961112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.961156 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.961205 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.983141 (+ 21936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.984303 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.985765 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.986102 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.986232 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.987838 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:54.987843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:54.987846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:54.987875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:54.989819 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.990432 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.990435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.990437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.990526 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:54.990850 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:54.990853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:54.990855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:54.991076 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:54.991208 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:54.991551 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 16:33:54.991671 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:33:54.991878 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.960594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43368)
0927 16:33:54.960883 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.960896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.961006 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.961336 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.961341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.982780 (+ 21439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.982948 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.982959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.984657 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.984660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.985611 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.985622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.986368 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.986371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.986478 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.986949 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.986963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:54.987439 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:54.989944 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.989951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.989954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.990320 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.990632 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:54.990635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:54.990639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:54.990743 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:54.991216 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:54.991221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:54.991348 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 16:33:54.991465 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 16:33:54.991629 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
Sep 27 16:33:54 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:33:55.003144 7229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:55.003316 7229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } }
I20260927 16:33:55.004418 7229 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969), 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61:44811)
I20260927 16:33:55.005039 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b0bbc7de2342e589dab032e20bea59" is_pre_election: true
I20260927 16:33:55.005056 6514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985d4c004a2f4febae194330d09193bb" is_pre_election: true
I20260927 16:33:55.005235 7107 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Handling vote request from an unknown peer 1ba3986e51f248c1b4c9dd5c4a724a9c
I20260927 16:33:55.005378 7107 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 1ba3986e51f248c1b4c9dd5c4a724a9c for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 16:33:55.005817 7172 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ba3986e51f248c1b4c9dd5c4a724a9c; no voters: 73b0bbc7de2342e589dab032e20bea59, 985d4c004a2f4febae194330d09193bb
I20260927 16:33:55.006021 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.996023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57446)
0927 16:33:54.996130 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:54.996144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:54.996218 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:54.996507 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:54.996510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:54.996691 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:54.996791 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.996798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.997782 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.997788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.998743 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:54.998754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.999324 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.999327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:54.999428 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:54.999910 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:54.999924 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.001426 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.003698 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.003702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.003704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.003931 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.004274 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.004277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.004279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.004363 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.004848 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.004851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.005615 (+ 764us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.005718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.005858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 16:33:55.006390 7229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:55.006618 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:54.996132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.0.98.60:33037)
0927 16:33:54.996233 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:54.996237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:54.996261 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:33:54.996326 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:54.996350 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:54.996398 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:54.996467 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:54.996910 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.997672 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.998871 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:54.999215 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:54.999310 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.001561 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.001565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.001568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.001593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.003580 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.004039 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.004042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.004044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.004177 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.004471 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.004474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.004476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.004723 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.004806 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.006356 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.006474 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:33:55.007679 7229 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 16:33:55.008421 6504 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33037: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33037
}
attrs {
promote: true
}
}
dest_uuid: "985d4c004a2f4febae194330d09193bb"
cas_config_opid_index: 5
I20260927 16:33:55.008968 6504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } }
I20260927 16:33:55.009954 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.011317 7064 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33037
I20260927 16:33:55.011478 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.011582 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.012931 7197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:33:55.013388 7069 master.cc:537] Master@127.0.98.60:33037 shutting down...
I20260927 16:33:55.014206 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:33:55.017081 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb -> Peer 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60:33037): Couldn't send request to peer 1ba3986e51f248c1b4c9dd5c4a724a9c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 16:33:55.020007 7069 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:55.020275 7069 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:55.020356 7069 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: stopping tablet replica
I20260927 16:33:55.020514 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.012034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.98.61:44811)
0927 16:33:55.012147 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.012152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.013846 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 16:33:55.014033 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.014037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.014098 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.014162 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.014863 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.015661 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.019042 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.019674 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.019762 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.019945 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.020304 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.020388 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:33:55.020928 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.011787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56700)
0927 16:33:55.013771 (+ 1984us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.013785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.013918 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.014615 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.014619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.014638 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.014730 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.014737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.015882 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.015886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.018887 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.018898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.020047 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.020051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.020142 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.020189 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.020693 (+ 504us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.020778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1909}
I20260927 16:33:55.021471 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:33:55.021924 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 16:33:55.024174 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.026450 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985d4c004a2f4febae194330d09193bb. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.026582 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:33:55.026669 7144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: true } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.026729 7144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:33:55.027801 7065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: attempting to promote NON_VOTER 73b0bbc7de2342e589dab032e20bea59 to VOTER
I20260927 16:33:55.028239 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } }
I20260927 16:33:55.029738 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.029334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57460)
0927 16:33:55.029468 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.029534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 16:33:55.029937 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 LEARNER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 16:33:55.030287 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 16:33:55.031461 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.030636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57462)
0927 16:33:55.031334 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.031384 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":80,"threads_started":1}
W20260927 16:33:55.031549 6490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb -> Peer 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60:33037): Couldn't send request to peer 1ba3986e51f248c1b4c9dd5c4a724a9c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:33:55.032081 7064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.033475 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.033571 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.033679 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.033746 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.034605 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.036042 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.036149 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:55.039127 7069 master.cc:552] Master@127.0.98.60:33037 shutdown complete.
I20260927 16:33:55.039196 7069 master_runner.cc:305] Clearing existing system tablet
I20260927 16:33:55.039485 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:55.039558 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:55.040848 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 16:33:55.041718 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.041985 7069 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:55.042104 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:55.042137 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "bd082ce4484dbc7aa2ede4438892fd44"
server_key_iv: "1d1864ad8c8671084d526bd68626fee5"
server_key_version: "encryptionkey@0"
I20260927 16:33:55.044092 7069 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:55.046324 7069 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:33:55.046407 7069 log.cc:1196] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 16:33:55.046635 7069 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Deleting consensus metadata
I20260927 16:33:55.046787 7069 master_runner.cc:315] Copying system tablet from 127.0.98.62:36969
I20260927 16:33:55.047842 7069 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36969
I20260927 16:33:55.057291 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.048702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.98.62:36969)
0927 16:33:55.048811 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.048816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.048831 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:33:55.048874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.048877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.048924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.049013 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.049571 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.050449 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.051667 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.052030 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.052139 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.053496 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.053500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.053504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.053533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.055388 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.055891 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.055894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.055897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.055989 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.056283 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.056286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.056288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.056613 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.056715 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.057001 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.057124 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:33:55.057535 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.048362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43374)
0927 16:33:55.048629 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.048645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.048756 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.049048 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.049051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.049297 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.049441 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.049448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.050570 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.050573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.051499 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.051509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.052161 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.052164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.052276 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.052888 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.052910 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.053336 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.055516 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.055519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.055521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.055784 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.056091 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.056094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.056096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.056190 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.056715 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.056718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.056822 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.056936 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.057088 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260927 16:33:55.058382 6524 tablet_copy_service.cc:140] P 985d4c004a2f4febae194330d09193bb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1ba3986e51f248c1b4c9dd5c4a724a9c ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43374)
I20260927 16:33:55.058492 6524 tablet_copy_service.cc:161] P 985d4c004a2f4febae194330d09193bb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1ba3986e51f248c1b4c9dd5c4a724a9c at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43374: session id = 1ba3986e51f248c1b4c9dd5c4a724a9c-00000000000000000000000000000000
I20260927 16:33:55.059571 6524 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:33:55.060937 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:33:55.063417 7069 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: tablet copy: Starting download of 0 data blocks...
I20260927 16:33:55.063598 7069 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: tablet copy: Starting download of 1 WAL segments...
I20260927 16:33:55.065186 7069 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:33:55.067085 6524 tablet_copy_service.cc:342] P 985d4c004a2f4febae194330d09193bb: Request end of tablet copy session 1ba3986e51f248c1b4c9dd5c4a724a9c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43374
I20260927 16:33:55.067179 6524 tablet_copy_service.cc:433] P 985d4c004a2f4febae194330d09193bb: ending tablet copy session 1ba3986e51f248c1b4c9dd5c4a724a9c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1ba3986e51f248c1b4c9dd5c4a724a9c
I20260927 16:33:55.074843 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:55.076758 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.076783 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.078508 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.082412 7069 server_base.cc:1034] running on GCE node
I20260927 16:33:55.082624 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:55.082960 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:55.084108 7069 hybrid_clock.cc:648] HybridClock initialized: now 1790526835084097 us; error 32 us; skew 500 ppm
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:55.085887 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 16:33:55.086414 7069 webserver.cc:460] Webserver started at http://127.0.98.60:34057/ using document root <none> and password file <none>
I20260927 16:33:55.086621 7069 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:55.086671 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:55.088212 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 16:33:55.089427 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.089721 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:55.089809 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "bd082ce4484dbc7aa2ede4438892fd44"
server_key_iv: "1d1864ad8c8671084d526bd68626fee5"
server_key_version: "encryptionkey@0"
I20260927 16:33:55.089910 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:55.109146 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:55.111606 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:55.116644 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33037
I20260927 16:33:55.116668 7323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33037 every 8 connection(s)
I20260927 16:33:55.117282 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:33:55.119172 7324 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:33:55.120266 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Bootstrap starting.
I20260927 16:33:55.131264 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:33:55.131578 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Bootstrap complete.
I20260927 16:33:55.132181 7324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } }
I20260927 16:33:55.132335 7324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ba3986e51f248c1b4c9dd5c4a724a9c, State: Initialized, Role: LEARNER
I20260927 16:33:55.132431 7324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } }
I20260927 16:33:55.132861 7324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:55.132845 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.133121 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: LEARNER
W20260927 16:33:55.135694 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:55.136063 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:55.136137 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:33:55.139750 7341 catalog_manager.cc:1259] Loaded cluster ID: 8c0f8f483f42473a8a0d38aff9c0e13b
I20260927 16:33:55.139812 7341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: loading cluster ID for follower catalog manager: success
I20260927 16:33:55.141588 7341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: acquiring CA information for follower catalog manager: success
W20260927 16:33:55.142701 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:55.142761 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:55.142884 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:55.142858 7341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:33:55.148939 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41789
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:55.150084 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:55.151182 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:33:55.156533 6829 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
W20260927 16:33:55.161434 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.161461 7249 server_base.cc:1034] running on GCE node
W20260927 16:33:55.161413 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.161408 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.162113 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:55.162840 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:55.164006 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790526835163974 us; error 45 us; skew 500 ppm
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:55.166776 7249 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:33:55.167874 7249 webserver.cc:460] Webserver started at http://127.0.98.1:41789/ using document root <none> and password file <none>
I20260927 16:33:55.168599 7249 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:55.168715 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:55.172814 7249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 16:33:55.175216 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.176602 7249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:33:55.176741 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "5ea7b9eb8f144161b922ae78202486be"
format_stamp: "Formatted at 2026-09-27 16:33:53 on dist-test-slave-t27c"
server_key: "11e02c552ed088f30e92416aae10b0c2"
server_key_iv: "6848bfc0086078ec9943589dc6df8ab4"
server_key_version: "encryptionkey@0"
I20260927 16:33:55.177314 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:55.190088 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:55.193085 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:55.193405 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:55.194365 7249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:55.195305 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:55.195358 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.195429 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:55.195459 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.205173 7249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39417
I20260927 16:33:55.205189 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39417 every 8 connection(s)
I20260927 16:33:55.206228 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:55.209271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7249
I20260927 16:33:55.209503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6700
I20260927 16:33:55.218356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41765
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36969
--webserver_port=38539
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:55.225951 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.210145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54311 (local address 127.0.98.61:44811)
0927 16:33:55.210334 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.210341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.210730 (+ 389us) server_negotiation.cc:407] Connection header received
0927 16:33:55.211184 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.211201 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.211273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.211340 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.213519 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.214126 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.216068 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.216347 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.216478 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.220845 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.220850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.220853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.220887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.223002 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.223868 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.223873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.223879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.223991 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.224484 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.224488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.224490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.224753 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.224844 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.225610 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.225751 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 16:33:55.227174 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.208771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.98.1:54311)
0927 16:33:55.210634 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.210646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.210928 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.211643 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.211650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.212419 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.213329 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.213347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.214396 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.214400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.215917 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.215929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.216531 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.216536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.216777 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.217611 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.217638 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.220622 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.223158 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.223165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.223169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.223738 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.224130 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.224137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.224142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.224336 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.224930 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.224934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.225274 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.226125 (+ 851us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.226309 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"mutex_wait_us":75,"spinlock_wait_cycles":36096,"thread_start_us":87,"threads_started":1}
I20260927 16:33:55.227476 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.208511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.98.1:58161)
0927 16:33:55.210348 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.210362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.210912 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.211639 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.211647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.212418 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.213331 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.213350 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.215782 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.215788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.216884 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.216898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.217606 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.217613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.217708 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.218265 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.218280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.220625 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.223130 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.223140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.223162 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.223750 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.224114 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.224120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.224125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.224316 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.224878 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.224887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.225272 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.225505 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.225918 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":148,"spinlock_wait_cycles":19328,"thread_start_us":164,"threads_started":1}
I20260927 16:33:55.228318 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.208138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58161 (local address 127.0.98.60:33037)
0927 16:33:55.208408 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.208412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.210480 (+ 2068us) server_negotiation.cc:407] Connection header received
0927 16:33:55.211259 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.211262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.211313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.211421 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.214370 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.215590 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.217094 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.217453 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.217590 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.220846 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.220852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.220856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.220892 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.222992 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.223893 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.223899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.223904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.224007 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.224459 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.224465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.224469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.224732 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.227815 (+ 3083us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.227923 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.228076 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260927 16:33:55.228631 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44811
I20260927 16:33:55.228890 7469 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.228943 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33037
I20260927 16:33:55.229014 7467 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.229336 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.208511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:46537)
0927 16:33:55.209747 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.209802 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.210912 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.217804 (+ 6892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.217809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.218118 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.218281 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.218289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.219348 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.219351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.220093 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.220106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.220892 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.220896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.221010 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.221666 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.221684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.223671 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.225994 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.225998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.226764 (+ 766us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.227020 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.227916 (+ 896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.227919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.227921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.228022 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.228749 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.228752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.228845 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.228971 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.229149 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":920,"threads_started":1}
I20260927 16:33:55.229434 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.208134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46537 (local address 127.0.98.62:36969)
0927 16:33:55.208283 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.208287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.217401 (+ 9114us) server_negotiation.cc:407] Connection header received
0927 16:33:55.217466 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.217472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.217541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.217630 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.218452 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.219234 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.220208 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.220601 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.220737 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.223870 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.223878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.223881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.223914 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.225854 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.227148 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.227151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.227153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.227251 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.228149 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.228152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.228154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.228377 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.228464 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.229132 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.229264 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:33:55.229522 7469 heartbeater.cc:511] Master 127.0.98.61:44811 requested a full tablet report, sending...
I20260927 16:33:55.230005 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:55.230093 7467 heartbeater.cc:511] Master 127.0.98.60:33037 requested a full tablet report, sending...
I20260927 16:33:55.230162 7468 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.230410 7468 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:55.231169 7288 ts_manager.cc:194] Registered new tserver with Master: 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417)
I20260927 16:33:55.231178 7099 ts_manager.cc:194] Registered new tserver with Master: 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417)
I20260927 16:33:55.232136 6504 ts_manager.cc:194] Re-registered known tserver with Master: 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417)
I20260927 16:33:55.232880 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46537
W20260927 16:33:55.338346 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:55.338629 7475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:55.338673 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:55.342723 7475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:55.342779 7475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:55.342839 7475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:55.348204 7475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38539
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:55.349201 7475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:55.350220 7475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:55.359745 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.359751 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.360074 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.360105 7475 server_base.cc:1034] running on GCE node
I20260927 16:33:55.360569 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:55.361223 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:55.362354 7475 hybrid_clock.cc:648] HybridClock initialized: now 1790526835362320 us; error 48 us; skew 500 ppm
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:55.364912 7475 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:33:55.365643 7475 webserver.cc:460] Webserver started at http://127.0.98.2:38539/ using document root <none> and password file <none>
I20260927 16:33:55.366120 7475 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:55.366245 7475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:55.370074 7475 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:55.372030 7490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.373245 7475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:55.373384 7475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "6b47527560b24b25974b6915e16d69ba"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "f6532c3a5c77bb396298b4638ad34c2f"
server_key_iv: "3903b1844c1213e44115c6daf033c6eb"
server_key_version: "encryptionkey@0"
I20260927 16:33:55.373921 7475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:55.380076 6963 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
I20260927 16:33:55.396417 7475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:55.399194 7475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:55.399345 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:55.399863 7475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:55.400645 7475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:55.400691 7475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.400729 7475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:55.400750 7475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.409405 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.405037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57468)
0927 16:33:55.405252 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.405271 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.405409 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.405777 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.405780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.405799 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.405922 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.405928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.406940 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.406945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.407909 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.407918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.408884 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.408891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.409008 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.409041 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.409127 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.409220 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 16:33:55.409426 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.405125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.98.60:33037)
0927 16:33:55.405243 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.405249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.405379 (+ 130us) server_negotiation.cc:407] Connection header received
0927 16:33:55.405547 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.405552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.405629 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.405748 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.406043 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.406793 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.408019 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.408738 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.408867 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.409076 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.409154 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.409255 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:33:55.410070 7290 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 16:33:55.410643 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985d4c004a2f4febae194330d09193bb. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.410796 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: LEARNER
I20260927 16:33:55.411038 7475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41765
I20260927 16:33:55.411127 7603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41765 every 8 connection(s)
I20260927 16:33:55.412050 7475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:55.418124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7475
I20260927 16:33:55.418336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6834
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:55.427490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36955
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36969
--webserver_port=34619
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:33:55.430909 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.413917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48003 (local address 127.0.98.61:44811)
0927 16:33:55.414068 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.414073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.415216 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 16:33:55.416222 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.416225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.416272 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.416374 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.418020 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.418889 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421085 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421529 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421625 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.425250 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.425257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.425263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.425298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.427784 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.428991 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.428995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.428997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429079 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429516 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.429519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.429521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429763 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.429831 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.430624 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.430745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:33:55.430909 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.415585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51827 (local address 127.0.98.62:36969)
0927 16:33:55.415715 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.415718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.416748 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 16:33:55.416927 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.416930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.416981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.417076 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.418014 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.418854 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421109 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421451 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421525 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.426044 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.426051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.426057 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.426096 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.428850 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429315 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.429319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.429321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429406 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429695 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.429698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.429700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429916 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.429981 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.430624 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.430745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:55.430997 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.414581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59161 (local address 127.0.98.60:33037)
0927 16:33:55.414716 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.414723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.416463 (+ 1740us) server_negotiation.cc:407] Connection header received
0927 16:33:55.416585 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.416589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.416638 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.416721 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.418013 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.418862 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421109 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421449 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421525 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.425471 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.425475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.425477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.425507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.428031 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.428963 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.428969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.428972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429068 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429535 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.429538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.429540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.429785 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.429862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.430626 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.430765 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:33:55.432789 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.414444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.98.2:59161)
0927 16:33:55.416358 (+ 1914us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.416369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.416477 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.416767 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.416773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.417261 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.417843 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.417862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.419040 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.419043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.420970 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.420980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421731 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.422068 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.423139 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.423158 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.425303 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.428592 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.428595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.428600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.428845 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.429189 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.429198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.429412 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.429942 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.429947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.430339 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.431143 (+ 804us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.431702 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"mutex_wait_us":799,"spinlock_wait_cycles":75136,"thread_start_us":160,"threads_started":1}
I20260927 16:33:55.433056 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.414937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:51827)
0927 16:33:55.416652 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.416662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.416806 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.417117 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.417119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.417335 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.417843 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.417862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.419040 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.419043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.420970 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.420980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421731 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.422065 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.423015 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.423032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.425879 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.428970 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.428974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.428976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.429207 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.429516 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.429521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.429597 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.430048 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.430050 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.430337 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.431142 (+ 805us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.431604 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1651,"mutex_wait_us":629,"spinlock_wait_cycles":23936,"thread_start_us":96,"threads_started":1}
I20260927 16:33:55.432788 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.414438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.98.2:48003)
0927 16:33:55.415045 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.415106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.416015 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.416609 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.416618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.417261 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.417842 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.417861 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.419389 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.419396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.420943 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.420956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.421811 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.421817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.422112 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.422969 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.422994 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.425051 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.428095 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.428103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.428124 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.428486 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.429200 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.429203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.429205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.429402 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.429925 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.429932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.430337 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.431718 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.431835 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":318,"spinlock_wait_cycles":61184,"thread_start_us":152,"threads_started":1}
I20260927 16:33:55.434116 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33037
I20260927 16:33:55.434211 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44811
I20260927 16:33:55.434379 7604 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.434381 7606 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.434928 7604 heartbeater.cc:511] Master 127.0.98.60:33037 requested a full tablet report, sending...
I20260927 16:33:55.434931 7606 heartbeater.cc:511] Master 127.0.98.61:44811 requested a full tablet report, sending...
I20260927 16:33:55.435461 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:55.435539 7605 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.435710 7605 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:55.436124 7099 ts_manager.cc:194] Registered new tserver with Master: 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765)
I20260927 16:33:55.436167 7288 ts_manager.cc:194] Registered new tserver with Master: 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765)
I20260927 16:33:55.436267 6504 ts_manager.cc:194] Re-registered known tserver with Master: 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765)
I20260927 16:33:55.436947 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51827
W20260927 16:33:55.541361 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:55.541669 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:55.541752 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:55.546363 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:55.546444 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:55.546531 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:55.551837 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34619
--tserver_master_addrs=127.0.98.62:36969,127.0.98.61:44811,127.0.98.60:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:55.552953 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:55.553934 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:55.563689 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:55.563717 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.563767 7611 server_base.cc:1034] running on GCE node
W20260927 16:33:55.563717 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:55.564260 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:55.564877 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:55.566028 7611 hybrid_clock.cc:648] HybridClock initialized: now 1790526835566014 us; error 25 us; skew 500 ppm
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:55.568840 7611 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:33:55.569752 7611 webserver.cc:460] Webserver started at http://127.0.98.3:34619/ using document root <none> and password file <none>
I20260927 16:33:55.570397 7611 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:55.570508 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:55.574558 7611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:33:55.576772 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.578135 7611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:55.578284 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "77c631d37446474a9fd23eac9594857a"
format_stamp: "Formatted at 2026-09-27 16:33:54 on dist-test-slave-t27c"
server_key: "96dc3614e32913ae9c33181be927c90f"
server_key_iv: "6875950aa7f1fad70b62d66d1d50cf6d"
server_key_version: "encryptionkey@0"
I20260927 16:33:55.578831 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:55.599382 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:55.602097 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:55.602324 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:55.602860 7611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:33:55.603636 7611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:55.603681 7611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.603720 7611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:55.603740 7611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:55.612831 7611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36955
I20260927 16:33:55.612849 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36955 every 8 connection(s)
I20260927 16:33:55.613920 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:33:55.624091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7611
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 16:33:55 dist-test-slave-t27c krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526835, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:55.631424 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.616442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:38353)
0927 16:33:55.618318 (+ 1876us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.618329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.618445 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.618710 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.618713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.619025 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.619603 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.619612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620687 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.620691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.621780 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.621785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.622458 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.622461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.622665 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.623401 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.623415 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.626112 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.628623 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.628626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.628628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.629706 (+ 1078us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.630212 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.630215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.630217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.630312 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.630738 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.630741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.630823 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.630906 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.631026 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1826,"mutex_wait_us":326,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260927 16:33:55.632067 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.615705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53897 (local address 127.0.98.61:44811)
0927 16:33:55.615829 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.615834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.617263 (+ 1429us) server_negotiation.cc:407] Connection header received
0927 16:33:55.618409 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.618413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.618461 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.618533 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.619734 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620259 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.621398 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.621618 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.621729 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.624679 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.624682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.624685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.624713 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.627817 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.628424 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.628428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.628430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.628522 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.628966 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.628970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.628972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.629208 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.629307 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.631786 (+ 2479us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.631908 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:55.632817 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.616610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38353 (local address 127.0.98.62:36969)
0927 16:33:55.616754 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.616759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.618412 (+ 1653us) server_negotiation.cc:407] Connection header received
0927 16:33:55.618528 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.618534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.618587 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.618676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.619748 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620551 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.621945 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.622320 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.622446 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.626538 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.626541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.626544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.626570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.628514 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.630001 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.630004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.630006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.630089 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.630417 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.630420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.630422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.630608 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.630697 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.632527 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.632646 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:33:55.634595 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36969
I20260927 16:33:55.634842 7741 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.634990 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.615704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56485 (local address 127.0.98.60:33037)
0927 16:33:55.615829 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.615834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.617321 (+ 1487us) server_negotiation.cc:407] Connection header received
0927 16:33:55.618100 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.618103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.618184 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.618273 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.619747 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620545 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.621919 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.622147 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.622273 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.626507 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.626510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.626513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.626539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.629365 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.632056 (+ 2691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.632059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.632061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.632152 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.633580 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.633583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.633585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.633798 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.633883 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.634713 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.634849 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:33:55.635423 7741 heartbeater.cc:511] Master 127.0.98.62:36969 requested a full tablet report, sending...
I20260927 16:33:55.631425 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.616156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.98.3:53897)
0927 16:33:55.617080 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.617136 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.617912 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.618565 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.618567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.619012 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.619590 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.619602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620397 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.620400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.621253 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.621271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.621763 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.621767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.621987 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.622667 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.622682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.624481 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.627925 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.627935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.627953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.628289 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.628623 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.628626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.628628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.628847 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.629362 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.629370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.629704 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.629935 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.630322 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":59,"spinlock_wait_cycles":13440,"thread_start_us":747,"threads_started":1}
I20260927 16:33:55.636329 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44811
I20260927 16:33:55.636418 7742 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.636602 7742 heartbeater.cc:511] Master 127.0.98.61:44811 requested a full tablet report, sending...
I20260927 16:33:55.636718 6504 ts_manager.cc:194] Re-registered known tserver with Master: 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:55.637360 7099 ts_manager.cc:194] Registered new tserver with Master: 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:55.637524 6504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38353
I20260927 16:33:55.638275 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.616157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.98.3:56485)
0927 16:33:55.617205 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.617239 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.617928 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.618449 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.618457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.619038 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:55.619606 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.619611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.620687 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.620691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.621774 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.621784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.622433 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.622438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.622624 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.623288 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:55.623299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:55.626386 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:55.629469 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.629473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.629475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.629901 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.632285 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:55.632288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:55.632290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:55.632369 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:55.633936 (+ 1567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:55.633939 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:55.634026 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.634416 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.634549 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":311,"spinlock_wait_cycles":24704,"thread_start_us":142,"threads_started":1}
I20260927 16:33:55.639230 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33037
I20260927 16:33:55.639303 7740 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:55.639479 7740 heartbeater.cc:511] Master 127.0.98.60:33037 requested a full tablet report, sending...
I20260927 16:33:55.640132 7288 ts_manager.cc:194] Registered new tserver with Master: 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:55.640333 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.626551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.98.61:44811)
0927 16:33:55.626775 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.626779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.627920 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 16:33:55.628084 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.628087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.628133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.628621 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.630149 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.630961 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.632087 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.632481 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.632569 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.636613 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.636620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.636623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.636647 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.638400 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.638861 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.638865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.638867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.638948 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.639243 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.639246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.639247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.639460 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.639594 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.639996 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.640120 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260927 16:33:55.642910 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.626335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.62:36969)
0927 16:33:55.628579 (+ 2244us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.628586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.628602 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:55.628637 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.628640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.628679 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.628753 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.629995 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.630838 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.633224 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.633636 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.633721 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.639276 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.639279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.639282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.639305 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.641114 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.641572 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.641578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.641581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.641671 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.641976 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.641979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.641983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.642145 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.642328 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.642544 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.642672 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1791,"thread_start_us":83,"threads_started":1}
I20260927 16:33:55.643306 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.627091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.60:33037)
0927 16:33:55.627735 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.627747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.629713 (+ 1966us) server_negotiation.cc:407] Connection header received
0927 16:33:55.629752 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.629755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.629801 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.629889 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.632648 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.633438 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.634570 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.634885 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.634978 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.636883 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.636886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.636889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.636909 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.641466 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.641963 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.641966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.641968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.642047 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.642371 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.642375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.642376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.642540 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.642678 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.642908 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.643148 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":87,"threads_started":1}
I20260927 16:33:55.645496 6504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:55.649333 6504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:55.651983 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: true } } }
I20260927 16:33:55.652093 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.655664 7759 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: attempting to promote NON_VOTER 1ba3986e51f248c1b4c9dd5c4a724a9c to VOTER
W20260927 16:33:55.656580 7065 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "73b0bbc7de2342e589dab032e20bea59" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260927 16:33:55.656773 7759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } }
I20260927 16:33:55.657866 7290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 LEARNER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:33:55.658581 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Refusing update from remote peer 985d4c004a2f4febae194330d09193bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:33:55.658979 7065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 16:33:55.659166 7758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 16:33:55.660844 7065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.661274 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.662233 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.662357 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.662693 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.662999 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.663093 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.663205 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.663259 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:55.663578 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.663678 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:55.664750 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "985d4c004a2f4febae194330d09193bb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:55.665097 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:55.671860 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.666353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41765 (local address 127.0.0.1:60144)
0927 16:33:55.666505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.666521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.666600 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.667445 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.667449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.667467 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.667582 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.667588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.669439 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.669446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.670299 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.670308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.671474 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.671478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.671568 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.671594 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.671642 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.671716 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 16:33:55.672633 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.666503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.98.2:41765)
0927 16:33:55.666877 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.666890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.666916 (+ 26us) server_negotiation.cc:407] Connection header received
0927 16:33:55.667006 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.667013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.667267 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.667955 (+ 688us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.668099 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.669354 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.670443 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.671321 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.672142 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.672299 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.672371 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.672482 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":133,"threads_started":1}
I20260927 16:33:55.672788 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.667703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.98.3:36955)
0927 16:33:55.668462 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.668474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.668496 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:33:55.668637 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.668642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.668830 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.668981 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.669155 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.670262 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.671415 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.672216 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.672334 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.672466 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.672558 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.672641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":113,"threads_started":1}
I20260927 16:33:55.672999 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.667220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36955 (local address 127.0.0.1:50384)
0927 16:33:55.667845 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.667866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.667944 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.668966 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.668969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.668986 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.669065 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.669070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.670379 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.670386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.671280 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.671290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.672334 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.672337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.672429 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.672450 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.672532 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.672604 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":92,"threads_started":1}
I20260927 16:33:55.673136 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.666908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39417 (local address 127.0.0.1:35572)
0927 16:33:55.667568 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.667584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.667679 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.668626 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.668629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.668645 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.668734 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.668739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.670233 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.670236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.671387 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.671400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.672650 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.672657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.672755 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.672781 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.672870 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.672935 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":101,"threads_started":1}
I20260927 16:33:55.673276 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.667093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.0.98.1:39417)
0927 16:33:55.667515 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.667526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.667997 (+ 471us) server_negotiation.cc:407] Connection header received
0927 16:33:55.668088 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.668093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.668315 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.668484 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.668864 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.670091 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.671521 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.672352 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.672465 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.672585 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.673064 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.673150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":102,"threads_started":1}
I20260927 16:33:55.675849 7538 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2d90aabbc4acf891d064384fc9435 (DEFAULT_TABLE table=test-table [id=f95809c673084606851b1d63e8c97700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:55.676093 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2d90aabbc4acf891d064384fc9435 (DEFAULT_TABLE table=test-table [id=f95809c673084606851b1d63e8c97700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:55.676247 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2d90aabbc4acf891d064384fc9435 (DEFAULT_TABLE table=test-table [id=f95809c673084606851b1d63e8c97700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:55.676893 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2d90aabbc4acf891d064384fc9435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:55.677088 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2d90aabbc4acf891d064384fc9435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:55.677088 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2d90aabbc4acf891d064384fc9435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:55.684182 7773 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Bootstrap starting.
I20260927 16:33:55.685412 7775 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Bootstrap starting.
I20260927 16:33:55.686487 7774 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Bootstrap starting.
I20260927 16:33:55.686519 7773 tablet_bootstrap.cc:654] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:55.687461 7775 tablet_bootstrap.cc:654] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:55.687726 7773 log.cc:824] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:55.688216 7775 log.cc:824] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:55.688457 7774 tablet_bootstrap.cc:654] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:55.689073 7774 log.cc:824] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:55.689707 7773 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: No bootstrap required, opened a new log
I20260927 16:33:55.689908 7773 ts_tablet_manager.cc:1397] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:33:55.691030 7774 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: No bootstrap required, opened a new log
I20260927 16:33:55.691201 7774 ts_tablet_manager.cc:1397] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:33:55.691701 7775 tablet_bootstrap.cc:492] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: No bootstrap required, opened a new log
I20260927 16:33:55.691857 7775 ts_tablet_manager.cc:1397] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 16:33:55.693815 7773 raft_consensus.cc:348] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.694031 7773 raft_consensus.cc:374] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:55.694063 7773 raft_consensus.cc:729] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b47527560b24b25974b6915e16d69ba, State: Initialized, Role: FOLLOWER
I20260927 16:33:55.694607 7773 consensus_queue.cc:260] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.695747 7773 ts_tablet_manager.cc:1428] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:33:55.695720 7774 raft_consensus.cc:348] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.696025 7605 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
I20260927 16:33:55.696066 7774 raft_consensus.cc:374] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:55.696133 7774 raft_consensus.cc:729] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea7b9eb8f144161b922ae78202486be, State: Initialized, Role: FOLLOWER
I20260927 16:33:55.696313 7775 raft_consensus.cc:348] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.696640 7775 raft_consensus.cc:374] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:55.696593 7774 consensus_queue.cc:260] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.696779 7775 raft_consensus.cc:729] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c631d37446474a9fd23eac9594857a, State: Initialized, Role: FOLLOWER
I20260927 16:33:55.697276 7775 consensus_queue.cc:260] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.697691 7774 ts_tablet_manager.cc:1428] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:33:55.698052 7468 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
I20260927 16:33:55.698405 7775 ts_tablet_manager.cc:1428] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260927 16:33:55.700371 7741 heartbeater.cc:503] Master 127.0.98.62:36969 was elected leader, sending a full tablet report...
W20260927 16:33:55.708148 7470 tablet.cc:2369] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:55.840754 7780 raft_consensus.cc:482] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:55.840977 7780 raft_consensus.cc:504] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.842331 7780 leader_election.cc:288] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765), 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:55.846383 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.842689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52175 (local address 127.0.98.3:36955)
0927 16:33:55.842826 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.842831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.842987 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:33:55.843145 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.843148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.843193 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.843274 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.843530 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.844208 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845395 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.845810 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845927 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.845988 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.846199 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.846265 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:33:55.846403 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.842580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36955 (local address 127.0.98.1:52175)
0927 16:33:55.842883 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.842900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.843007 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.843299 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.843302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.843319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.843421 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.843426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.844314 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.844320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.845267 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.845279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.845940 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.846036 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.846065 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.846160 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.846257 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":25,"thread_start_us":107,"threads_started":1}
I20260927 16:33:55.846483 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.842689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46439 (local address 127.0.98.2:41765)
0927 16:33:55.842881 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.842886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.843226 (+ 340us) server_negotiation.cc:407] Connection header received
0927 16:33:55.843391 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.843394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.843444 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.843528 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:55.843784 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.844361 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845480 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.845874 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845993 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.846052 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.846274 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.846341 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 16:33:55.847004 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.842580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41765 (local address 127.0.98.1:46439)
0927 16:33:55.843139 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:55.843150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:55.843276 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:55.843538 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:55.843541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:55.843557 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:55.843679 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.843684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.844476 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.844479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.845364 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:55.845372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.845994 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.845998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:55.846092 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.846115 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:33:55.846253 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:33:55.846333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":314,"thread_start_us":78,"threads_started":1}
I20260927 16:33:55.847157 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b2d90aabbc4acf891d064384fc9435" candidate_uuid: "5ea7b9eb8f144161b922ae78202486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c631d37446474a9fd23eac9594857a" is_pre_election: true
I20260927 16:33:55.847505 7694 raft_consensus.cc:2454] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea7b9eb8f144161b922ae78202486be in term 0.
I20260927 16:33:55.847654 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b2d90aabbc4acf891d064384fc9435" candidate_uuid: "5ea7b9eb8f144161b922ae78202486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b47527560b24b25974b6915e16d69ba" is_pre_election: true
I20260927 16:33:55.847937 7558 raft_consensus.cc:2454] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea7b9eb8f144161b922ae78202486be in term 0.
I20260927 16:33:55.848026 7355 leader_election.cc:302] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea7b9eb8f144161b922ae78202486be, 77c631d37446474a9fd23eac9594857a; no voters:
I20260927 16:33:55.848320 7780 raft_consensus.cc:2781] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:55.848412 7780 raft_consensus.cc:482] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:55.848443 7780 raft_consensus.cc:3037] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:55.849445 7780 raft_consensus.cc:504] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.849781 7780 leader_election.cc:288] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [CANDIDATE]: Term 1 election: Requested vote from peers 6b47527560b24b25974b6915e16d69ba (127.0.98.2:41765), 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:55.850121 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b2d90aabbc4acf891d064384fc9435" candidate_uuid: "5ea7b9eb8f144161b922ae78202486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b47527560b24b25974b6915e16d69ba"
I20260927 16:33:55.850118 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b2d90aabbc4acf891d064384fc9435" candidate_uuid: "5ea7b9eb8f144161b922ae78202486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c631d37446474a9fd23eac9594857a"
I20260927 16:33:55.850263 7558 raft_consensus.cc:3037] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:55.850263 7694 raft_consensus.cc:3037] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:55.851507 7558 raft_consensus.cc:2454] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea7b9eb8f144161b922ae78202486be in term 1.
I20260927 16:33:55.851507 7694 raft_consensus.cc:2454] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea7b9eb8f144161b922ae78202486be in term 1.
I20260927 16:33:55.851765 7356 leader_election.cc:302] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea7b9eb8f144161b922ae78202486be, 6b47527560b24b25974b6915e16d69ba; no voters:
I20260927 16:33:55.851910 7780 raft_consensus.cc:2781] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:55.852097 7780 raft_consensus.cc:686] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 1 LEADER]: Becoming Leader. State: Replica: 5ea7b9eb8f144161b922ae78202486be, State: Running, Role: LEADER
I20260927 16:33:55.852458 7780 consensus_queue.cc:237] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:55.856436 6502 catalog_manager.cc:5495] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea7b9eb8f144161b922ae78202486be (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5ea7b9eb8f144161b922ae78202486be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } health_report { overall_health: UNKNOWN } } }
W20260927 16:33:55.865759 7743 tablet.cc:2369] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:55.905711 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.902390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.0.98.1:39417)
0927 16:33:55.902519 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.902522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.902535 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:55.902609 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.902611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.902682 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.902775 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:55.903003 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.903496 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.904304 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.904530 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.904609 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.904899 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:55.905382 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:55.905499 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.905549 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.905601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:33:55.911475 7694 raft_consensus.cc:1261] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Refusing update from remote peer 5ea7b9eb8f144161b922ae78202486be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:55.911475 7558 raft_consensus.cc:1261] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Refusing update from remote peer 5ea7b9eb8f144161b922ae78202486be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:55.912196 7784 consensus_queue.cc:1035] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:55.912413 7780 consensus_queue.cc:1035] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:33:55.914069 7607 tablet.cc:2369] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:33:55.922574 7788 mvcc.cc:205] Tried to move back new op lower bound from 7333997919886127104 to 7333997919655366656. Current Snapshot: MvccSnapshot[applied={T|T < 7333997919886127104 or (T in {7333997919886127104})}]
I20260927 16:33:55.925467 7789 mvcc.cc:205] Tried to move back new op lower bound from 7333997919886127104 to 7333997919655366656. Current Snapshot: MvccSnapshot[applied={T|T < 7333997919886127104}]
I20260927 16:33:55.931457 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:55.931619 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:55.936187 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 985d4c004a2f4febae194330d09193bb: Sending DeleteTablet for 3 replicas of tablet 37b2d90aabbc4acf891d064384fc9435
I20260927 16:33:55.936968 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b2d90aabbc4acf891d064384fc9435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60144
I20260927 16:33:55.937531 7797 tablet_replica.cc:330] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: stopping tablet replica
I20260927 16:33:55.937973 7797 raft_consensus.cc:2229] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:55.938004 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b2d90aabbc4acf891d064384fc9435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35572
I20260927 16:33:55.938480 7797 raft_consensus.cc:2258] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:55.938506 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b2d90aabbc4acf891d064384fc9435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50384
I20260927 16:33:55.939076 7800 tablet_replica.cc:330] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: stopping tablet replica
I20260927 16:33:55.939389 7800 raft_consensus.cc:2229] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:55.939699 7800 raft_consensus.cc:2258] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:55.940366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6472
I20260927 16:33:55.940608 7797 ts_tablet_manager.cc:1905] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:55.941494 7800 ts_tablet_manager.cc:1905] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:55.944128 7800 ts_tablet_manager.cc:1918] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:55.944228 7800 log.cc:1196] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/37b2d90aabbc4acf891d064384fc9435
I20260927 16:33:55.944346 7797 ts_tablet_manager.cc:1918] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:55.944530 7797 log.cc:1196] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/37b2d90aabbc4acf891d064384fc9435
I20260927 16:33:55.944556 7800 ts_tablet_manager.cc:1939] T 37b2d90aabbc4acf891d064384fc9435 P 77c631d37446474a9fd23eac9594857a: Deleting consensus metadata
I20260927 16:33:55.944941 7797 ts_tablet_manager.cc:1939] T 37b2d90aabbc4acf891d064384fc9435 P 6b47527560b24b25974b6915e16d69ba: Deleting consensus metadata
I20260927 16:33:55.946393 7799 tablet_replica.cc:330] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: stopping tablet replica
I20260927 16:33:55.946702 7799 raft_consensus.cc:2229] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:33:55.947057 7799 raft_consensus.cc:2258] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:33:55.948635 7627 connection.cc:535] server connection from 127.0.0.1:50384 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:33:55.948838 7799 ts_tablet_manager.cc:1905] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:33:55.949222 7491 connection.cc:535] server connection from 127.0.0.1:60144 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:33:55.951431 7799 ts_tablet_manager.cc:1918] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:55.951521 7799 log.cc:1196] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/37b2d90aabbc4acf891d064384fc9435
I20260927 16:33:55.951830 7799 ts_tablet_manager.cc:1939] T 37b2d90aabbc4acf891d064384fc9435 P 5ea7b9eb8f144161b922ae78202486be: Deleting consensus metadata
W20260927 16:33:55.952836 7354 connection.cc:441] server connection from 127.0.0.1:35572 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35572 (request call id 1) could send its response
I20260927 16:33:55.956928 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.939342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.98.61:44811)
0927 16:33:55.939459 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.939464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.947519 (+ 8055us) server_negotiation.cc:407] Connection header received
0927 16:33:55.947636 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.947642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.947709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.947797 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.949030 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.949813 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.951125 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.951499 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.951586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.953637 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.953641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.953644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.953670 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.955288 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.955708 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.955714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.955715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.955795 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.956073 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.956078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.956079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.956249 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.956383 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.956610 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.956723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 16:33:55.963824 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.953619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.98.60:33037)
0927 16:33:55.953722 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.953726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.954077 (+ 351us) server_negotiation.cc:407] Connection header received
0927 16:33:55.954279 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.954282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.954333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.954383 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.955562 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.956355 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.958027 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.958454 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.958527 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.960272 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.960276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.960279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.960304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.961975 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.962489 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.962493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.962495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.962573 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.962900 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.962904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.962905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.963101 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.963202 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.963534 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.963651 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260927 16:33:55.966063 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.939085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.0.98.60:33037)
0927 16:33:55.939191 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.939195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.957024 (+ 17829us) server_negotiation.cc:407] Connection header received
0927 16:33:55.957193 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.957195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.957246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.957295 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.958099 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.958908 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.960047 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.960379 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.960467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.961958 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.961962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.961964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.961988 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.963952 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.964655 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.964658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.964659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.964736 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.965043 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.965047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.965049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.965228 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.965335 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.965816 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.965922 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:33:55.967792 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:55.954162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.0.98.61:44811)
0927 16:33:55.954313 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:55.954319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:55.958701 (+ 4382us) server_negotiation.cc:407] Connection header received
0927 16:33:55.958838 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:55.958841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:55.958877 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:55.958924 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:55.960002 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.960723 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.961969 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:55.962393 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:55.962478 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:55.964177 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:55.964181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:55.964183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:55.964208 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:55.966054 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.966699 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.966702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.966704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.966788 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:55.967120 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:55.967122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:55.967123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:55.967276 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:55.967375 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:55.967589 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 16:33:55.967665 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 16:33:56.924202 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:56.923583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:43791)
0927 16:33:56.923956 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:56.924084 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20260927 16:33:56.924679 7605 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:56.928356 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:56.927712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:38559)
0927 16:33:56.928139 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:56.928230 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":73,"threads_started":1}
W20260927 16:33:56.928663 7468 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:56.928812 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:56.928343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:55543)
0927 16:33:56.928628 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:56.928712 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
W20260927 16:33:56.929035 7741 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:57.436971 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 985d4c004a2f4febae194330d09193bb)
I20260927 16:33:57.437119 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } }
I20260927 16:33:57.437495 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 985d4c004a2f4febae194330d09193bb)
I20260927 16:33:57.437683 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } }
I20260927 16:33:57.437748 7810 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969), 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61:44811)
I20260927 16:33:57.438459 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.437877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43880)
0927 16:33:57.438274 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.438361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":69,"threads_started":1}
W20260927 16:33:57.438668 7275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:57.439160 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969), 1ba3986e51f248c1b4c9dd5c4a724a9c (127.0.98.60:33037)
I20260927 16:33:57.439359 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.438917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43896)
0927 16:33:57.439193 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.439272 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":51,"threads_started":1}
W20260927 16:33:57.439512 7092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:57.440117 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.439869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43908)
0927 16:33:57.440008 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.440055 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 16:33:57.440349 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.440137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43920)
0927 16:33:57.440252 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.440293 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 16:33:57.440327 7275 leader_election.cc:334] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
W20260927 16:33:57.440461 7092 leader_election.cc:334] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:57.442538 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.438072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.98.61:44811)
0927 16:33:57.438347 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.438354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.438395 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:33:57.439138 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.439141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.439203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.439285 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.439658 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.440441 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.441519 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.441937 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.442053 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.442115 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.442346 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.442412 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260927 16:33:57.442551 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.437973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44811 (local address 127.0.0.1:56736)
0927 16:33:57.438315 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.438331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.438455 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.439408 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.439412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.439434 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.439530 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.439536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.440568 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.440572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.441383 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.441392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.442054 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.442057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.442213 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.442237 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.442331 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.442405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260927 16:33:57.442863 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.438978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33037 (local address 127.0.0.1:57504)
0927 16:33:57.439366 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.439379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.439473 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.439822 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.439825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.439842 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.439929 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.439937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.440915 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.440918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.441680 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.441688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.442421 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.442425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.442534 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.442551 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.442642 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.442744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":74,"threads_started":1}
I20260927 16:33:57.442956 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "73b0bbc7de2342e589dab032e20bea59" is_pre_election: true
I20260927 16:33:57.443099 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ba3986e51f248c1b4c9dd5c4a724a9c in term 1.
I20260927 16:33:57.443271 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.439084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.60:33037)
0927 16:33:57.439390 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.439472 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.439488 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:57.439600 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.439605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.439649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.439720 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.440050 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.440801 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.441814 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.442311 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.442421 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.442472 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.442896 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.443116 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260927 16:33:57.443378 7273 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ba3986e51f248c1b4c9dd5c4a724a9c, 73b0bbc7de2342e589dab032e20bea59; no voters: 985d4c004a2f4febae194330d09193bb
I20260927 16:33:57.443552 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 16:33:57.443611 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 985d4c004a2f4febae194330d09193bb)
I20260927 16:33:57.443671 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:33:57.444705 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } }
I20260927 16:33:57.445065 7810 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 election: Requested vote from peers 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969), 73b0bbc7de2342e589dab032e20bea59 (127.0.98.61:44811)
I20260927 16:33:57.445304 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73b0bbc7de2342e589dab032e20bea59" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" is_pre_election: true
I20260927 16:33:57.445408 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.445131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43926)
0927 16:33:57.445295 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.445352 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 16:33:57.445402 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "73b0bbc7de2342e589dab032e20bea59"
I20260927 16:33:57.445495 7115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:33:57.445564 7290 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73b0bbc7de2342e589dab032e20bea59 in current term 2: Already voted for candidate 1ba3986e51f248c1b4c9dd5c4a724a9c in this term.
I20260927 16:33:57.445921 7091 leader_election.cc:302] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73b0bbc7de2342e589dab032e20bea59; no voters: 1ba3986e51f248c1b4c9dd5c4a724a9c, 985d4c004a2f4febae194330d09193bb
I20260927 16:33:57.446038 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.445823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43934)
0927 16:33:57.445903 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.445951 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 16:33:57.446126 7275 leader_election.cc:334] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
I20260927 16:33:57.446712 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ba3986e51f248c1b4c9dd5c4a724a9c in term 2.
I20260927 16:33:57.446818 7811 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:33:57.446998 7273 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ba3986e51f248c1b4c9dd5c4a724a9c, 73b0bbc7de2342e589dab032e20bea59; no voters: 985d4c004a2f4febae194330d09193bb
I20260927 16:33:57.447157 7810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:33:57.447342 7810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [term 2 LEADER]: Becoming Leader. State: Replica: 1ba3986e51f248c1b4c9dd5c4a724a9c, State: Running, Role: LEADER
I20260927 16:33:57.447583 7810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } }
I20260927 16:33:57.448889 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba3986e51f248c1b4c9dd5c4a724a9c. Latest consensus state: current_term: 2 leader_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:57.448995 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:57.449258 7822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:57.451457 7822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:57.451850 7822 catalog_manager.cc:1259] Loaded cluster ID: 8c0f8f483f42473a8a0d38aff9c0e13b
I20260927 16:33:57.451918 7822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:57.452435 7822 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:57.452761 7822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Loaded TSK: 0
I20260927 16:33:57.453364 7822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:33:57.481587 7288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:57.484978 7288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:57.487884 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.487587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43948)
0927 16:33:57.487734 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.487805 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 16:33:57.487879 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [term 2 FOLLOWER]: Refusing update from remote peer 1ba3986e51f248c1b4c9dd5c4a724a9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 16:33:57.488458 7810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 16:33:57.488595 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.488375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:43960)
0927 16:33:57.488485 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.488537 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:33:57.488722 7275 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c -> Peer 985d4c004a2f4febae194330d09193bb (127.0.98.62:36969): Couldn't send request to peer 985d4c004a2f4febae194330d09193bb. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:33:57.489946 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba3986e51f248c1b4c9dd5c4a724a9c. Latest consensus state: current_term: 2 leader_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:57.490094 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:57.491248 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:57.491391 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:57.491632 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "985d4c004a2f4febae194330d09193bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } peers { permanent_uuid: "73b0bbc7de2342e589dab032e20bea59" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44811 } attrs { promote: false } } peers { permanent_uuid: "1ba3986e51f248c1b4c9dd5c4a724a9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33037 } attrs { promote: false } } }
I20260927 16:33:57.491750 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b0bbc7de2342e589dab032e20bea59 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:33:57.501283 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.497091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.98.1:39417)
0927 16:33:57.497361 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.497365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.497376 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:33:57.497419 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.497421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.497459 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.497864 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.497974 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.498764 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.500117 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.500734 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.500980 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.501049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.501089 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.501152 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260927 16:33:57.501281 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.496939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39417 (local address 127.0.0.1:35586)
0927 16:33:57.497056 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.497072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.497160 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.497607 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.497610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.497627 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.497751 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.497757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.498881 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.498884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.499976 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.499983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.500855 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.500859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.500993 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.501015 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.501079 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.501146 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":18}
I20260927 16:33:57.501654 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.497522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41765 (local address 127.0.0.1:60150)
0927 16:33:57.497608 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.497624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.497701 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.498423 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.498426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.498442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.498516 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.498521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.499527 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.499531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.500357 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.500404 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.501285 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.501289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.501369 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.501398 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.501467 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.501536 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":25}
I20260927 16:33:57.502066 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.497932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.0.98.2:41765)
0927 16:33:57.498186 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.498191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.498206 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:33:57.498259 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.498262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.498310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.498402 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.498642 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.499414 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.500534 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.501156 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.501289 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.501644 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.501687 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.501920 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260927 16:33:57.502259 7401 tablet_service.cc:1467] Processing CreateTablet for tablet f7aff24d13f44a85acbf3dde68375020 (DEFAULT_TABLE table=test-table [id=1a622396a38e404cbf0d9cad08e03798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:57.502576 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aff24d13f44a85acbf3dde68375020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:57.502578 7538 tablet_service.cc:1467] Processing CreateTablet for tablet f7aff24d13f44a85acbf3dde68375020 (DEFAULT_TABLE table=test-table [id=1a622396a38e404cbf0d9cad08e03798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:57.502846 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aff24d13f44a85acbf3dde68375020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:57.502847 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.0.98.3:36955)
0927 16:33:57.498230 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.498234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.498587 (+ 353us) server_negotiation.cc:407] Connection header received
0927 16:33:57.498684 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.498687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.498732 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.498828 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.499092 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.499688 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.501629 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.502245 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.502357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.502419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.502652 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.502731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":64,"threads_started":1}
I20260927 16:33:57.503089 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.497522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36955 (local address 127.0.0.1:50398)
0927 16:33:57.498490 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.498502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.498591 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.498834 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.498837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.498852 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.498985 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.498990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.499797 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.499800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.501521 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.501529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.502356 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.502361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.502515 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.502539 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.502623 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.502824 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":292,"thread_start_us":101,"threads_started":1}
I20260927 16:33:57.504734 7832 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: Bootstrap starting.
I20260927 16:33:57.505080 7674 tablet_service.cc:1467] Processing CreateTablet for tablet f7aff24d13f44a85acbf3dde68375020 (DEFAULT_TABLE table=test-table [id=1a622396a38e404cbf0d9cad08e03798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:57.505336 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aff24d13f44a85acbf3dde68375020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:57.505623 7832 tablet_bootstrap.cc:654] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:57.506846 7832 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: No bootstrap required, opened a new log
I20260927 16:33:57.506939 7832 ts_tablet_manager.cc:1397] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:57.507333 7832 raft_consensus.cc:348] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.507467 7832 raft_consensus.cc:374] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:57.507509 7832 raft_consensus.cc:729] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea7b9eb8f144161b922ae78202486be, State: Initialized, Role: FOLLOWER
I20260927 16:33:57.507634 7832 consensus_queue.cc:260] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.507866 7835 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Bootstrap starting.
I20260927 16:33:57.507972 7832 ts_tablet_manager.cc:1428] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:57.508082 7833 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba: Bootstrap starting.
I20260927 16:33:57.508936 7835 tablet_bootstrap.cc:654] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:57.509910 7833 tablet_bootstrap.cc:654] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:57.510129 7835 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: No bootstrap required, opened a new log
I20260927 16:33:57.510241 7835 ts_tablet_manager.cc:1397] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:57.510548 7467 heartbeater.cc:503] Master 127.0.98.60:33037 was elected leader, sending a full tablet report...
I20260927 16:33:57.510664 7835 raft_consensus.cc:348] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.510784 7835 raft_consensus.cc:374] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:57.510820 7835 raft_consensus.cc:729] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c631d37446474a9fd23eac9594857a, State: Initialized, Role: FOLLOWER
I20260927 16:33:57.510946 7835 consensus_queue.cc:260] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.511247 7835 ts_tablet_manager.cc:1428] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:57.512249 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.511686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:37049)
0927 16:33:57.512075 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.512137 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":85,"threads_started":1}
I20260927 16:33:57.512385 7833 tablet_bootstrap.cc:492] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba: No bootstrap required, opened a new log
I20260927 16:33:57.512466 7833 ts_tablet_manager.cc:1397] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 16:33:57.512684 7740 heartbeater.cc:503] Master 127.0.98.60:33037 was elected leader, sending a full tablet report...
I20260927 16:33:57.512894 7833 raft_consensus.cc:348] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.513013 7833 raft_consensus.cc:374] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:57.513055 7833 raft_consensus.cc:729] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b47527560b24b25974b6915e16d69ba, State: Initialized, Role: FOLLOWER
I20260927 16:33:57.513182 7833 consensus_queue.cc:260] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.513520 7833 ts_tablet_manager.cc:1428] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:33:57.514365 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.513971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:45809)
0927 16:33:57.514104 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.514154 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 16:33:57.514420 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.508704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:43925)
0927 16:33:57.514259 (+ 5555us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.514316 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5443,"thread_start_us":89,"threads_started":1}
I20260927 16:33:57.514767 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.513901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:42403)
0927 16:33:57.514616 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.514675 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":83,"threads_started":1}
I20260927 16:33:57.515224 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.514810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:36331)
0927 16:33:57.515089 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.515142 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260927 16:33:57.515532 7604 heartbeater.cc:503] Master 127.0.98.60:33037 was elected leader, sending a full tablet report...
I20260927 16:33:57.515539 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.515143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:42487)
0927 16:33:57.515397 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.515469 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260927 16:33:57.541059 7842 raft_consensus.cc:482] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:33:57.541224 7842 raft_consensus.cc:504] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.542429 7842 leader_election.cc:288] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417), 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:57.546449 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39417 (local address 127.0.98.2:39359)
0927 16:33:57.542816 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.542833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.542941 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.543215 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.543218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.543246 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.543370 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.543376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.544329 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.544332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.545282 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.545291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.546002 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.546005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.546120 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.546138 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.546246 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.546317 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":2304}
I20260927 16:33:57.546449 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.542803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39359 (local address 127.0.98.1:39417)
0927 16:33:57.542928 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.542934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.542952 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:33:57.543043 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.543048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.543103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.543200 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.543481 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.544220 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.545417 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.545858 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.545980 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.546049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.546263 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.546337 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:33:57.546641 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.543073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40545 (local address 127.0.98.3:36955)
0927 16:33:57.543234 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.543239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.543255 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:33:57.543314 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.543316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.543388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.543456 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:57.543771 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.544391 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.545556 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.546073 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.546213 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.546271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.546461 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.546530 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:33:57.547009 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.542956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36955 (local address 127.0.98.2:40545)
0927 16:33:57.543136 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.543149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:57.543235 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:57.543515 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:57.543518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:57.543534 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:57.543618 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.543623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.544533 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.544537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.545429 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:57.545438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.546218 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.546222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:57.546326 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.546354 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:33:57.546444 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:33:57.546513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 16:33:57.547123 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aff24d13f44a85acbf3dde68375020" candidate_uuid: "6b47527560b24b25974b6915e16d69ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea7b9eb8f144161b922ae78202486be" is_pre_election: true
I20260927 16:33:57.547399 7421 raft_consensus.cc:2454] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b47527560b24b25974b6915e16d69ba in term 0.
I20260927 16:33:57.547533 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aff24d13f44a85acbf3dde68375020" candidate_uuid: "6b47527560b24b25974b6915e16d69ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c631d37446474a9fd23eac9594857a" is_pre_election: true
I20260927 16:33:57.547683 7694 raft_consensus.cc:2454] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b47527560b24b25974b6915e16d69ba in term 0.
I20260927 16:33:57.547859 7492 leader_election.cc:302] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea7b9eb8f144161b922ae78202486be, 6b47527560b24b25974b6915e16d69ba; no voters:
I20260927 16:33:57.548120 7842 raft_consensus.cc:2781] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:33:57.548180 7842 raft_consensus.cc:482] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:33:57.548237 7842 raft_consensus.cc:3037] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:57.549245 7842 raft_consensus.cc:504] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.549578 7842 leader_election.cc:288] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [CANDIDATE]: Term 1 election: Requested vote from peers 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417), 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955)
I20260927 16:33:57.549894 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aff24d13f44a85acbf3dde68375020" candidate_uuid: "6b47527560b24b25974b6915e16d69ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea7b9eb8f144161b922ae78202486be"
I20260927 16:33:57.549943 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aff24d13f44a85acbf3dde68375020" candidate_uuid: "6b47527560b24b25974b6915e16d69ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c631d37446474a9fd23eac9594857a"
I20260927 16:33:57.550020 7421 raft_consensus.cc:3037] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:57.550071 7694 raft_consensus.cc:3037] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:57.551048 7694 raft_consensus.cc:2454] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b47527560b24b25974b6915e16d69ba in term 1.
I20260927 16:33:57.551088 7421 raft_consensus.cc:2454] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b47527560b24b25974b6915e16d69ba in term 1.
I20260927 16:33:57.551328 7492 leader_election.cc:302] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b47527560b24b25974b6915e16d69ba, 77c631d37446474a9fd23eac9594857a; no voters:
I20260927 16:33:57.551499 7842 raft_consensus.cc:2781] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:57.551734 7842 raft_consensus.cc:686] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [term 1 LEADER]: Becoming Leader. State: Replica: 6b47527560b24b25974b6915e16d69ba, State: Running, Role: LEADER
I20260927 16:33:57.552186 7842 consensus_queue.cc:237] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } }
I20260927 16:33:57.552454 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.552225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:42157)
0927 16:33:57.552338 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.552394 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 16:33:57.554615 7288 catalog_manager.cc:5495] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b47527560b24b25974b6915e16d69ba (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b47527560b24b25974b6915e16d69ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b47527560b24b25974b6915e16d69ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 } health_report { overall_health: UNKNOWN } } }
I20260927 16:33:57.580718 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.577186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.98.2:41765)
0927 16:33:57.577316 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:57.577324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:57.577337 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:33:57.577472 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:57.577475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:57.577531 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:57.577625 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:33:57.577814 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.578429 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.579353 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:57.579561 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:57.579642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:57.579863 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:33:57.580316 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:33:57.580432 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:57.580481 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:33:57.580547 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:33:57.583711 7694 raft_consensus.cc:1261] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Refusing update from remote peer 6b47527560b24b25974b6915e16d69ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:57.583946 7421 raft_consensus.cc:1261] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Refusing update from remote peer 6b47527560b24b25974b6915e16d69ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:33:57.584244 7842 consensus_queue.cc:1035] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c631d37446474a9fd23eac9594857a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:57.584527 7846 consensus_queue.cc:1035] T f7aff24d13f44a85acbf3dde68375020 P 6b47527560b24b25974b6915e16d69ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea7b9eb8f144161b922ae78202486be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:33:57.586196 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.585883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:56519)
0927 16:33:57.586040 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.586097 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 16:33:57.586720 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.586414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:55791)
0927 16:33:57.586581 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.586639 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 16:33:57.589648 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.589385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.3:56581)
0927 16:33:57.589525 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.589578 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 16:33:57.590672 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.589521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.2:51891)
0927 16:33:57.590527 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.590582 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":925}
I20260927 16:33:57.592975 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.592717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.98.1:47723)
0927 16:33:57.592851 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.592906 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 16:33:57.598052 7288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:57.598241 7288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:33:57.601743 7288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba3986e51f248c1b4c9dd5c4a724a9c: Sending DeleteTablet for 3 replicas of tablet f7aff24d13f44a85acbf3dde68375020
I20260927 16:33:57.602454 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aff24d13f44a85acbf3dde68375020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50398
I20260927 16:33:57.602535 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aff24d13f44a85acbf3dde68375020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35586
I20260927 16:33:57.602557 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aff24d13f44a85acbf3dde68375020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:33:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60150
I20260927 16:33:57.602954 7859 tablet_replica.cc:330] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: stopping tablet replica
I20260927 16:33:57.603086 7859 raft_consensus.cc:2229] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:57.603101 7860 tablet_replica.cc:330] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be: stopping tablet replica
I20260927 16:33:57.603223 7859 raft_consensus.cc:2258] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:57.603371 7860 raft_consensus.cc:2229] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:33:57.603539 7860 raft_consensus.cc:2258] T f7aff24d13f44a85acbf3dde68375020 P 5ea7b9eb8f144161b922ae78202486be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:33:57.603883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7249
I20260927 16:33:57.604866 7859 ts_tablet_manager.cc:1905] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:33:57.606901 7859 ts_tablet_manager.cc:1918] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:33:57.606963 7859 log.cc:1196] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/f7aff24d13f44a85acbf3dde68375020
I20260927 16:33:57.607172 7859 ts_tablet_manager.cc:1939] T f7aff24d13f44a85acbf3dde68375020 P 77c631d37446474a9fd23eac9594857a: Deleting consensus metadata
I20260927 16:33:57.607779 7273 catalog_manager.cc:4841] TS 77c631d37446474a9fd23eac9594857a (127.0.98.3:36955): tablet f7aff24d13f44a85acbf3dde68375020 (table test-table [id=1a622396a38e404cbf0d9cad08e03798]) successfully deleted
I20260927 16:33:57.613376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7475
I20260927 16:33:57.613440 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.613174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39417 (local address 127.0.0.1:35602)
0927 16:33:57.613297 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.613351 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 16:33:57.613636 7273 catalog_manager.cc:4584] TS 5ea7b9eb8f144161b922ae78202486be (127.0.98.1:39417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7aff24d13f44a85acbf3dde68375020: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39417: connect: Connection refused (error 111)
I20260927 16:33:57.620673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7611
I20260927 16:33:57.620934 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:57.620645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41765 (local address 127.0.0.1:60162)
0927 16:33:57.620776 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:57.620838 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:33:57.626322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 6968
I20260927 16:33:57.632076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7069
2026-09-27T16:33:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5342 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:33:57 dist-test-slave-t27c krb5kdc[7864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:33:57 dist-test-slave-t27c krb5kdc[7864](info): set up 2 sockets
Sep 27 16:33:57 dist-test-slave-t27c krb5kdc[7864](info): commencing operation
krb5kdc: starting...
W20260927 16:33:58.678876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:33:58 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:33:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 16:33:58.805771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:58.920461 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:58.920746 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:58.920787 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:58.924862 7877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:33:58.924933 7877 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:33:58.924952 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:58.924969 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:58.924986 7877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:33:58.925001 7877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:33:58.930382 7877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43665
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:58.931490 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:58.932497 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:58.941522 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:58.941593 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:58.941604 7877 server_base.cc:1034] running on GCE node
W20260927 16:33:58.941522 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:58.942548 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:58.943794 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:58.944954 7877 hybrid_clock.cc:648] HybridClock initialized: now 1790526838944918 us; error 49 us; skew 500 ppm
Sep 27 16:33:58 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:58.947808 7877 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 16:33:58.948787 7877 webserver.cc:460] Webserver started at http://127.0.98.62:34793/ using document root <none> and password file <none>
I20260927 16:33:58.949373 7877 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:58.949491 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:58.949733 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:58.952064 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "97627fd83775479b8f5701ee01d860ea"
format_stamp: "Formatted at 2026-09-27 16:33:58 on dist-test-slave-t27c"
server_key: "39cc56ce1ca3844b2dd4ee9cae9e7117"
server_key_iv: "fdb2b59bdb48945a94edb268b2365f68"
server_key_version: "encryptionkey@0"
I20260927 16:33:58.952688 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "97627fd83775479b8f5701ee01d860ea"
format_stamp: "Formatted at 2026-09-27 16:33:58 on dist-test-slave-t27c"
server_key: "39cc56ce1ca3844b2dd4ee9cae9e7117"
server_key_iv: "fdb2b59bdb48945a94edb268b2365f68"
server_key_version: "encryptionkey@0"
I20260927 16:33:58.956650 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 16:33:58.959132 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:58.960366 7877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:33:58.960505 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "97627fd83775479b8f5701ee01d860ea"
format_stamp: "Formatted at 2026-09-27 16:33:58 on dist-test-slave-t27c"
server_key: "39cc56ce1ca3844b2dd4ee9cae9e7117"
server_key_iv: "fdb2b59bdb48945a94edb268b2365f68"
server_key_version: "encryptionkey@0"
I20260927 16:33:58.960642 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:58.972102 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:58.975224 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:58.975435 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:58.983843 7877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43665
I20260927 16:33:58.983860 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43665 every 8 connection(s)
I20260927 16:33:58.985138 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:33:58.988111 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:58.992563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7877
I20260927 16:33:58.992713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:33:58.992990 392 external_mini_cluster.cc:1428] Setting key 13e67ce43689ae6107fec4b684b45b3d
I20260927 16:33:58.994364 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Bootstrap starting.
I20260927 16:33:58.996646 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:58.997383 7955 log.cc:824] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:58.999296 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: No bootstrap required, opened a new log
Sep 27 16:33:58 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:59.002961 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } }
I20260927 16:33:59.003273 7955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:59.003356 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97627fd83775479b8f5701ee01d860ea, State: Initialized, Role: FOLLOWER
I20260927 16:33:59.003870 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } }
I20260927 16:33:59.004017 7955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:33:59.004108 7955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:33:59.004204 7955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:33:59.005056 7955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } }
I20260927 16:33:59.005396 7955 leader_election.cc:302] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97627fd83775479b8f5701ee01d860ea; no voters:
I20260927 16:33:59.005721 7955 leader_election.cc:288] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:33:59.005806 7960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:33:59.006040 7960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [term 1 LEADER]: Becoming Leader. State: Replica: 97627fd83775479b8f5701ee01d860ea, State: Running, Role: LEADER
I20260927 16:33:59.006361 7960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } }
I20260927 16:33:59.007068 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:33:59.007459 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97627fd83775479b8f5701ee01d860ea. Latest consensus state: current_term: 1 leader_uuid: "97627fd83775479b8f5701ee01d860ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } } }
I20260927 16:33:59.007583 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:59.008172 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:58.994538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.0.98.62:43665)
0927 16:33:58.994980 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:58.994994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:58.995031 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:33:58.995692 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:58.995709 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:58.996043 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:58.996405 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:58.997218 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:58.998494 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:58.999328 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:58.999847 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.000016 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.002554 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.002560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.002573 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.002599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.004514 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.004935 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.004941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.004947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.005078 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.005359 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.005362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.005364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.005883 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.006104 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.006678 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.006972 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":134,"threads_started":1}
I20260927 16:33:59.008174 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97627fd83775479b8f5701ee01d860ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97627fd83775479b8f5701ee01d860ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43665 } } }
I20260927 16:33:59.008428 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea [sys.catalog]: This master's current role is: LEADER
I20260927 16:33:59.008590 7964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:33:59.011840 7964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:33:59.017552 7964 catalog_manager.cc:1347] Generated new cluster ID: 35da14a17129488da70b6540009482dd
I20260927 16:33:59.017643 7964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:33:59.055577 7964 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:33:59.056703 7964 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:33:59.063655 7964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Generated new TSK 0
I20260927 16:33:59.064504 7964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 16:33:59.101745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43665
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:59.217664 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:59.218036 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:59.218118 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:59.222373 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:59.222493 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:59.222568 7981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:59.222651 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:33:59.228091 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:59.229231 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:59.230317 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:59.239842 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:59.239854 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:59.239848 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:59.240290 7981 server_base.cc:1034] running on GCE node
I20260927 16:33:59.240835 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:59.241539 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:59.242719 7981 hybrid_clock.cc:648] HybridClock initialized: now 1790526839242705 us; error 26 us; skew 500 ppm
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:59.245671 7981 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:33:59.246639 7981 webserver.cc:460] Webserver started at http://127.0.98.1:46599/ using document root <none> and password file <none>
I20260927 16:33:59.247202 7981 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:59.247298 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:59.247534 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:59.249625 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "4909aa5e91619297b64f0e8e58c886dd"
server_key_iv: "dee49911622f92797df0c16aa0e7f661"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.250265 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "4909aa5e91619297b64f0e8e58c886dd"
server_key_iv: "dee49911622f92797df0c16aa0e7f661"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.254045 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:33:59.256651 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.258008 7981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:33:59.258160 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "4909aa5e91619297b64f0e8e58c886dd"
server_key_iv: "dee49911622f92797df0c16aa0e7f661"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.258345 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:59.278477 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:59.281844 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:59.282049 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:59.283670 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:59.283749 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.283841 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:59.283890 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:59.295008 7981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44735
I20260927 16:33:59.295096 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44735 every 8 connection(s)
I20260927 16:33:59.296120 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:33:59.298571 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.284578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54415 (local address 127.0.98.62:43665)
0927 16:33:59.284709 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.284713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.285743 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 16:33:59.286886 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.286892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.286958 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.287056 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.288998 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.289898 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.291511 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.291816 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.291942 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.294914 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.294918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.294921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.294951 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.296491 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.297139 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.297143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.297145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.297236 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.297493 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.297494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.297496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.297671 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.297740 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.298298 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.298429 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:33:59.298766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 7981
I20260927 16:33:59.298892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:33:59.299216 392 external_mini_cluster.cc:1428] Setting key 63238074bb4bb8bd9c6524a472e2acf7
I20260927 16:33:59.299674 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.285109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.1:54415)
0927 16:33:59.285585 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.285631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.286610 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.287562 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.287574 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.288080 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.288786 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.288806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.290044 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.290050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.291386 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.291396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.291980 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.291983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.292168 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.292849 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.292871 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.294723 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.296651 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.296657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.296672 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.297007 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.297303 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.297305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.297306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.297433 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.297756 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.297761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.298006 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.298239 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.298567 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260927 16:33:59.301156 8112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43665
I20260927 16:33:59.301312 8112 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:59.301914 8112 heartbeater.cc:511] Master 127.0.98.62:43665 requested a full tablet report, sending...
I20260927 16:33:59.303753 7908 ts_manager.cc:194] Registered new tserver with Master: 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735)
I20260927 16:33:59.305245 7908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54415
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:33:59.311995 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.303502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52751 (local address 127.0.98.62:43665)
0927 16:33:59.303626 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.303630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.303902 (+ 272us) server_negotiation.cc:407] Connection header received
0927 16:33:59.304114 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.304117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.304162 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.304250 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.304842 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.305558 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.307033 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.307343 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.307453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.308914 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.308919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.308922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.308949 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.310497 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.310964 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.310967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.310968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.311055 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.311345 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.311348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.311349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.311516 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.311600 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.311789 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.311880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:33:59.312064 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.303418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.1:52751)
0927 16:33:59.303814 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.303832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.304000 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.304287 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.304293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.304540 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.304712 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.304723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.305706 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.305709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.306885 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.306898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.307477 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.307481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.307595 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.308239 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.308274 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.308764 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.310676 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.310679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.310680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.310872 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.311169 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.311171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.311173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.311254 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.311603 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.311605 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.311679 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.311756 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.311865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 16:33:59.329701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43665
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:59.441250 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:59.441526 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:59.441571 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:59.445652 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:59.445719 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:59.445758 8121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:59.445852 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:33:59.451184 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:59.452281 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:59.453331 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:59.463671 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:59.463768 8121 server_base.cc:1034] running on GCE node
W20260927 16:33:59.463671 8129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:59.463666 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:59.464351 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:59.465118 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:59.466250 8121 hybrid_clock.cc:648] HybridClock initialized: now 1790526839466210 us; error 45 us; skew 500 ppm
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:59.468993 8121 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:33:59.469940 8121 webserver.cc:460] Webserver started at http://127.0.98.2:38455/ using document root <none> and password file <none>
I20260927 16:33:59.470585 8121 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:59.470678 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:59.470911 8121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:59.473009 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "076e1a3e1f31506ca64339068f539c2f"
server_key_iv: "c1f7f27e7fa3a4175469e7244f9fa80e"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.473598 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "076e1a3e1f31506ca64339068f539c2f"
server_key_iv: "c1f7f27e7fa3a4175469e7244f9fa80e"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.477378 8121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:33:59.479848 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.480983 8121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:33:59.481130 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "076e1a3e1f31506ca64339068f539c2f"
server_key_iv: "c1f7f27e7fa3a4175469e7244f9fa80e"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.481242 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:59.498955 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:59.502270 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:59.502509 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:59.504053 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:59.504127 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.504195 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:59.504244 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:59.515820 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260927 16:33:59.515923 8251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260927 16:33:59.516937 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:33:59.518262 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.504923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46219 (local address 127.0.98.62:43665)
0927 16:33:59.505160 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.505165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.506061 (+ 896us) server_negotiation.cc:407] Connection header received
0927 16:33:59.507197 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.507204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.507270 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.507328 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.509091 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.509755 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.511070 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.511293 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.511407 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.514372 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.514379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.514385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.514420 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.516101 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.516607 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.516610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.516612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.516704 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.517028 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.517032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.517033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.517262 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.517366 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.517967 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.518094 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 16:33:59.519002 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.505359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.2:46219)
0927 16:33:59.505909 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.505952 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.506945 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.507814 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.507821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.508298 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.508949 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.508964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.509873 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.509877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.510945 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.510954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.511458 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.511462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.511686 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.512326 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.512349 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.514132 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.516222 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.516227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.516238 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.516487 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.516791 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.516793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.516795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.516914 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.517384 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.517392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.517689 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.517867 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.518109 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260927 16:33:59.520143 8252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43665
I20260927 16:33:59.520298 8252 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:59.520787 8252 heartbeater.cc:511] Master 127.0.98.62:43665 requested a full tablet report, sending...
I20260927 16:33:59.521792 7907 ts_manager.cc:194] Registered new tserver with Master: a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:33:59.522583 7907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46219
I20260927 16:33:59.526252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8121
I20260927 16:33:59.526382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:33:59.526664 392 external_mini_cluster.cc:1428] Setting key 2d443014351b7a468c69132ca579b605
I20260927 16:33:59.529153 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.522203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.2:49773)
0927 16:33:59.522553 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.522565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.522643 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.522903 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.522909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.523111 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.523200 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.523206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.523935 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.523943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.524488 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.524495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.524924 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.524928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.525020 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.525536 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.525548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.525839 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.527681 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.527686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.527691 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.527954 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.528244 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.528246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.528247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.528318 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.528790 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.528792 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.528859 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.528934 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.529028 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
I20260927 16:33:59.529376 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.522279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49773 (local address 127.0.98.62:43665)
0927 16:33:59.522425 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.522429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.522638 (+ 209us) server_negotiation.cc:407] Connection header received
0927 16:33:59.522747 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.522750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.522791 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.522873 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.523312 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.523813 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.524601 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.524818 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.524922 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.525980 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.525984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.525988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.526013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.527579 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.528060 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.528064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.528065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.528154 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.528404 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.528407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.528408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.528692 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.528804 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.529024 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.529150 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:33:59.554546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43665
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:33:59.666929 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:33:59.667248 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:33:59.667294 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:33:59.671501 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:33:59.671566 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:33:59.671607 8261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:33:59.671639 8261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:33:59.676963 8261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:33:59.677991 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:33:59.679064 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:33:59.689085 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:59.689124 8261 server_base.cc:1034] running on GCE node
W20260927 16:33:59.689306 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:33:59.689425 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:33:59.689824 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:33:59.690546 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:33:59.691701 8261 hybrid_clock.cc:648] HybridClock initialized: now 1790526839691684 us; error 31 us; skew 500 ppm
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:33:59.694466 8261 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:33:59.695340 8261 webserver.cc:460] Webserver started at http://127.0.98.3:37653/ using document root <none> and password file <none>
I20260927 16:33:59.695875 8261 fs_manager.cc:359] Metadata directory not provided
I20260927 16:33:59.695935 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:33:59.696105 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:33:59.698331 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "176385af8648417a8336fd6d80ecf638"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "aeb0084c80bdbbd67cd2603d2ca5579e"
server_key_iv: "fcf78930ac6d217d56531aaa06e8163b"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.698916 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "176385af8648417a8336fd6d80ecf638"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "aeb0084c80bdbbd67cd2603d2ca5579e"
server_key_iv: "fcf78930ac6d217d56531aaa06e8163b"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.703191 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 16:33:59.705582 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.706923 8261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:33:59.707056 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "176385af8648417a8336fd6d80ecf638"
format_stamp: "Formatted at 2026-09-27 16:33:59 on dist-test-slave-t27c"
server_key: "aeb0084c80bdbbd67cd2603d2ca5579e"
server_key_iv: "fcf78930ac6d217d56531aaa06e8163b"
server_key_version: "encryptionkey@0"
I20260927 16:33:59.707170 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:33:59.738835 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:33:59.742453 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:33:59.742666 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:33:59.744139 8261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:33:59.744215 8261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:33:59.744284 8261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:33:59.744335 8261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:59.755064 8261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37441
I20260927 16:33:59.755189 8391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37441 every 8 connection(s)
I20260927 16:33:59.756183 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:33:59.758658 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.745249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60825 (local address 127.0.98.62:43665)
0927 16:33:59.745411 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.745416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.746289 (+ 873us) server_negotiation.cc:407] Connection header received
0927 16:33:59.747501 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.747505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.747570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.747674 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.749272 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.750228 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.751675 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.751994 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.752098 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.754962 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.754965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.754967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.755020 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.756661 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.757259 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.757262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.757263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.757342 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.757605 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.757607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.757608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.757780 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.757854 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.758422 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.758536 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:33:59.759471 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.745633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.3:60825)
0927 16:33:59.746098 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.746149 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.747253 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.747857 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.747865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.748338 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.749095 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.749117 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.750378 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.750382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.751536 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.751546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.752136 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.752140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.752324 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.752999 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.753021 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.754733 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.756790 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.756795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.756809 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.757159 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.757407 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.757409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.757410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.757528 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.757873 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.757877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.758108 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.758347 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.758606 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260927 16:33:59.760749 8392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43665
I20260927 16:33:59.760936 8392 heartbeater.cc:464] Registering TS with master...
I20260927 16:33:59.761252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8261
I20260927 16:33:59.761380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:33:59.761547 8392 heartbeater.cc:511] Master 127.0.98.62:43665 requested a full tablet report, sending...
I20260927 16:33:59.761751 392 external_mini_cluster.cc:1428] Setting key 849a2266aa9791fc56f84a17068f7db4
I20260927 16:33:59.762681 7907 ts_manager.cc:194] Registered new tserver with Master: 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441)
I20260927 16:33:59.763471 7907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60825
I20260927 16:33:59.763969 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:33:59.771438 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.763378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41447 (local address 127.0.98.62:43665)
0927 16:33:59.763496 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.763503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.763518 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:33:59.763560 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.763563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.763599 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.763682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.764242 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.764987 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.766329 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.766718 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.766839 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.767964 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.767968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.767971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.767994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.769683 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.770149 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.770153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.770155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.770286 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.770583 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.770588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.770589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.770769 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.770888 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.771146 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.771256 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:33:59.771451 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.763073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.98.3:41447)
0927 16:33:59.763351 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.763367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.763461 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.763713 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.763716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.763942 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:33:59.764120 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.764128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.765142 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.765148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.766124 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.766133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.766834 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.766837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.766911 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.767498 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:33:59.767509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:33:59.767819 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:33:59.769824 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.769827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.769836 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.770042 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.770389 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:33:59.770394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:33:59.770396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:33:59.770488 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:33:59.770877 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:33:59.770880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:33:59.770982 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.771091 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.771245 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:33:59.788530 7958 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:33:59.782359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.0.98.62:43665)
0927 16:33:59.782497 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.782500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.782528 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:33:59.782572 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.782574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.782614 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.782690 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.783377 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.783897 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.784866 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.785082 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.785164 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.788369 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50388: BlockingRecv error: recv got EOF from 127.0.0.1:50388 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 16:33:59 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:33:59.797885 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.790089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.98.62:43665)
0927 16:33:59.790239 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.790243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.790398 (+ 155us) server_negotiation.cc:407] Connection header received
0927 16:33:59.790496 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.790498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.790551 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.790617 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:33:59.791278 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.791776 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.792535 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.792739 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.792812 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.795072 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:33:59.795075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:33:59.795077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:33:59.795099 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:33:59.796487 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.796839 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.796841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.796842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.796910 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:33:59.797122 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:33:59.797124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:33:59.797125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:33:59.797283 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:33:59.797378 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.797586 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.797674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:33:59.800704 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:33:59.803903 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:33:59.818715 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.812768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44735 (local address 127.0.0.1:56892)
0927 16:33:59.813557 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.813607 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.813741 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.814483 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.814486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.814499 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:59.814579 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.814588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.815967 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.815970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.817240 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.817249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.818086 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.818090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.818238 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.818282 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.818400 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.818543 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":66,"thread_start_us":86,"threads_started":1}
I20260927 16:33:59.818884 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.813329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.98.3:37441)
0927 16:33:59.813855 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.813869 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.813887 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:33:59.813960 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.813965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.814192 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.814687 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:59.814815 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.815897 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.817404 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.818294 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.818399 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.818570 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.818623 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.818720 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":103,"threads_started":1}
I20260927 16:33:59.819200 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.812902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.1:44735)
0927 16:33:59.813616 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.813623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.813883 (+ 260us) server_negotiation.cc:407] Connection header received
0927 16:33:59.813960 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.813965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.814207 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.814343 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:59.814692 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.815839 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.817365 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.817937 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.818087 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.818281 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.818953 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.819063 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":80,"threads_started":1}
I20260927 16:33:59.818727 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.812884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37441 (local address 127.0.0.1:46142)
0927 16:33:59.813763 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.813776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.813860 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.814360 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.814365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.814395 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:59.814534 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.814540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.816025 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.816029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.817259 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.817270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.818418 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.818421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.818499 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.818522 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.818581 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.818635 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":63,"spinlock_wait_cycles":896,"thread_start_us":88,"threads_started":1}
I20260927 16:33:59.820434 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.813811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:41868)
0927 16:33:59.814284 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 16:33:59.814301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:33:59.814385 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:33:59.815307 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:33:59.815310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:33:59.815328 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:33:59.815421 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.815428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.816897 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.816900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.818008 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:33:59.818019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.820014 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.820017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:33:59.820131 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.820157 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:33:59.820240 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:33:59.820310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260927 16:33:59.820533 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:33:59.814202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.0.98.2:43717)
0927 16:33:59.814842 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 16:33:59.814856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:33:59.814880 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:33:59.814948 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:33:59.814952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:33:59.815173 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:33:59.815320 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:33:59.815672 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.816774 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.818163 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:33:59.819673 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:33:59.819780 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:33:59.819943 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 16:33:59.820298 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 16:33:59.820395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":99,"threads_started":1}
I20260927 16:33:59.822346 8326 tablet_service.cc:1467] Processing CreateTablet for tablet d57077b2e3014da6a8052902dd6b283e (DEFAULT_TABLE table=test-table [id=110312da2937458ba807ffedeac6c6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:59.822346 8046 tablet_service.cc:1467] Processing CreateTablet for tablet d57077b2e3014da6a8052902dd6b283e (DEFAULT_TABLE table=test-table [id=110312da2937458ba807ffedeac6c6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:59.823395 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57077b2e3014da6a8052902dd6b283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:59.823411 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57077b2e3014da6a8052902dd6b283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:59.823305 8178 tablet_service.cc:1467] Processing CreateTablet for tablet d57077b2e3014da6a8052902dd6b283e (DEFAULT_TABLE table=test-table [id=110312da2937458ba807ffedeac6c6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:33:59.824169 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57077b2e3014da6a8052902dd6b283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:33:59.830135 8420 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Bootstrap starting.
I20260927 16:33:59.831310 8421 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Bootstrap starting.
I20260927 16:33:59.832125 8422 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Bootstrap starting.
I20260927 16:33:59.832351 8420 tablet_bootstrap.cc:654] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:59.833076 8420 log.cc:824] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:59.833340 8421 tablet_bootstrap.cc:654] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:59.834048 8422 tablet_bootstrap.cc:654] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:33:59.834128 8421 log.cc:824] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:59.834856 8422 log.cc:824] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:33:59.835206 8420 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: No bootstrap required, opened a new log
I20260927 16:33:59.835381 8420 ts_tablet_manager.cc:1397] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 16:33:59.835994 8421 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: No bootstrap required, opened a new log
I20260927 16:33:59.836175 8421 ts_tablet_manager.cc:1397] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:33:59.836448 8422 tablet_bootstrap.cc:492] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: No bootstrap required, opened a new log
I20260927 16:33:59.836627 8422 ts_tablet_manager.cc:1397] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 16:33:59.839495 8422 raft_consensus.cc:348] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.839493 8420 raft_consensus.cc:348] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.839749 8420 raft_consensus.cc:374] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:59.839751 8422 raft_consensus.cc:374] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:59.839809 8420 raft_consensus.cc:729] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Initialized, Role: FOLLOWER
I20260927 16:33:59.839813 8422 raft_consensus.cc:729] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Initialized, Role: FOLLOWER
I20260927 16:33:59.840242 8422 consensus_queue.cc:260] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.840283 8420 consensus_queue.cc:260] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.840904 8252 heartbeater.cc:503] Master 127.0.98.62:43665 was elected leader, sending a full tablet report...
I20260927 16:33:59.841313 8422 ts_tablet_manager.cc:1428] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 16:33:59.841332 8420 ts_tablet_manager.cc:1428] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:33:59.841639 8392 heartbeater.cc:503] Master 127.0.98.62:43665 was elected leader, sending a full tablet report...
I20260927 16:33:59.841708 8421 raft_consensus.cc:348] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.841960 8421 raft_consensus.cc:374] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:33:59.842017 8421 raft_consensus.cc:729] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Initialized, Role: FOLLOWER
I20260927 16:33:59.842453 8421 consensus_queue.cc:260] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:33:59.843093 8112 heartbeater.cc:503] Master 127.0.98.62:43665 was elected leader, sending a full tablet report...
I20260927 16:33:59.843338 8421 ts_tablet_manager.cc:1428] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 16:34:00.007347 8393 tablet.cc:2369] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:34:00.018272 8253 tablet.cc:2369] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:34:00.047483 8113 tablet.cc:2369] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:34:00.072566 8428 raft_consensus.cc:482] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:00.072773 8428 raft_consensus.cc:504] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:34:00.073899 8428 leader_election.cc:288] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.079191 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.074832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56325 (local address 127.0.98.2:43717)
0927 16:34:00.074960 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.074982 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.075128 (+ 146us) server_negotiation.cc:407] Connection header received
0927 16:34:00.075225 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.075227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.075288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.075366 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.075590 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.076302 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.077881 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.078504 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.078614 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.078678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.078982 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.079061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:34:00.079191 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.074316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45017 (local address 127.0.98.3:37441)
0927 16:34:00.074446 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.074450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.074677 (+ 227us) server_negotiation.cc:407] Connection header received
0927 16:34:00.074849 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.074855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.074916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.075015 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.075324 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.076140 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.077882 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.078507 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.078614 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.078678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.078982 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.079061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:34:00.079226 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.074216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37441 (local address 127.0.98.1:45017)
0927 16:34:00.074569 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.074588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.074724 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.075047 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.075050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.075074 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.075215 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.075225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.076286 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.076289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.077744 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.077756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.078652 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.078656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.078783 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.078840 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.078927 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.079035 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":57,"thread_start_us":166,"threads_started":1}
I20260927 16:34:00.079228 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.074716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.1:56325)
0927 16:34:00.075034 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.075047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.075135 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.075389 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.075392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.075407 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.075490 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.075494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.076447 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.076450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.077747 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.077756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.078652 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.078656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.078837 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.078854 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.078927 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.079038 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":130,"thread_start_us":81,"threads_started":1}
I20260927 16:34:00.079952 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57077b2e3014da6a8052902dd6b283e" candidate_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176385af8648417a8336fd6d80ecf638" is_pre_election: true
I20260927 16:34:00.079964 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57077b2e3014da6a8052902dd6b283e" candidate_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" is_pre_election: true
I20260927 16:34:00.080245 8206 raft_consensus.cc:2454] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09af8e633b2e46e9a5409a7e9c5338fc in term 0.
I20260927 16:34:00.080255 8346 raft_consensus.cc:2454] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09af8e633b2e46e9a5409a7e9c5338fc in term 0.
I20260927 16:34:00.080729 7997 leader_election.cc:302] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.081036 8428 raft_consensus.cc:2781] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:00.081127 8428 raft_consensus.cc:482] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:00.081171 8428 raft_consensus.cc:3037] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.082329 8428 raft_consensus.cc:504] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:34:00.082674 8428 leader_election.cc:288] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [CANDIDATE]: Term 1 election: Requested vote from peers 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.083060 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57077b2e3014da6a8052902dd6b283e" candidate_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176385af8648417a8336fd6d80ecf638"
I20260927 16:34:00.083060 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57077b2e3014da6a8052902dd6b283e" candidate_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
I20260927 16:34:00.083186 8206 raft_consensus.cc:3037] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.083192 8346 raft_consensus.cc:3037] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.084565 8206 raft_consensus.cc:2454] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09af8e633b2e46e9a5409a7e9c5338fc in term 1.
I20260927 16:34:00.084908 7997 leader_election.cc:302] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.085036 8346 raft_consensus.cc:2454] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09af8e633b2e46e9a5409a7e9c5338fc in term 1.
I20260927 16:34:00.085182 8428 raft_consensus.cc:2781] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:00.085562 8428 raft_consensus.cc:686] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 LEADER]: Becoming Leader. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Running, Role: LEADER
I20260927 16:34:00.086356 8428 consensus_queue.cc:237] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } }
I20260927 16:34:00.089869 7908 catalog_manager.cc:5495] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } }
I20260927 16:34:00.109699 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.106052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.0.98.1:44735)
0927 16:34:00.106179 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.106183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.106196 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:34:00.106333 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.106336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.106404 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.106584 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.106846 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.107346 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.108276 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.108479 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.108557 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.108835 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.109277 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.109366 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.109454 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.109517 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:34:00.115888 8346 raft_consensus.cc:1261] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Refusing update from remote peer 09af8e633b2e46e9a5409a7e9c5338fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.115918 8206 raft_consensus.cc:1261] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Refusing update from remote peer 09af8e633b2e46e9a5409a7e9c5338fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.116622 8431 consensus_queue.cc:1035] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.116830 8428 consensus_queue.cc:1035] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.132808 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.132995 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.135388 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Sending DeleteTablet for 3 replicas of tablet d57077b2e3014da6a8052902dd6b283e
I20260927 16:34:00.136139 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet d57077b2e3014da6a8052902dd6b283e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260927 16:34:00.136304 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet d57077b2e3014da6a8052902dd6b283e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 16:34:00.136777 8444 tablet_replica.cc:330] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: stopping tablet replica
I20260927 16:34:00.136760 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet d57077b2e3014da6a8052902dd6b283e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260927 16:34:00.137091 8444 raft_consensus.cc:2229] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:34:00.137202 8446 tablet_replica.cc:330] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: stopping tablet replica
I20260927 16:34:00.137506 8446 raft_consensus.cc:2229] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.137543 8444 raft_consensus.cc:2258] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.137619 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:34:00.137804 8446 raft_consensus.cc:2258] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.137889 8445 tablet_replica.cc:330] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: stopping tablet replica
W20260927 16:34:00.138043 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:34:00.138150 8445 raft_consensus.cc:2229] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.138473 8445 raft_consensus.cc:2258] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.139475 8446 ts_tablet_manager.cc:1905] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.139879 8445 ts_tablet_manager.cc:1905] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.142557 8446 ts_tablet_manager.cc:1918] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.142668 8446 log.cc:1196] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/d57077b2e3014da6a8052902dd6b283e
I20260927 16:34:00.143000 8446 ts_tablet_manager.cc:1939] T d57077b2e3014da6a8052902dd6b283e P 176385af8648417a8336fd6d80ecf638: Deleting consensus metadata
I20260927 16:34:00.143689 8444 ts_tablet_manager.cc:1905] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.143785 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (DEFAULT_TABLE table=test-table [id=bfba26459cc040f7af39cbe2ac772dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.144011 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (DEFAULT_TABLE table=test-table [id=bfba26459cc040f7af39cbe2ac772dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.144256 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.145735 8445 ts_tablet_manager.cc:1918] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.145833 8445 log.cc:1196] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/d57077b2e3014da6a8052902dd6b283e
I20260927 16:34:00.146142 8445 ts_tablet_manager.cc:1939] T d57077b2e3014da6a8052902dd6b283e P a88fadc568a043fc89e8b0a95b60e3a4: Deleting consensus metadata
I20260927 16:34:00.146642 8420 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Bootstrap starting.
I20260927 16:34:00.147687 8420 tablet_bootstrap.cc:654] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.147710 7893 catalog_manager.cc:4841] TS a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717): tablet d57077b2e3014da6a8052902dd6b283e (table test-table [id=110312da2937458ba807ffedeac6c6d9]) successfully deleted
I20260927 16:34:00.148100 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.148239 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (DEFAULT_TABLE table=test-table [id=bfba26459cc040f7af39cbe2ac772dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.148496 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.149259 7893 catalog_manager.cc:4841] TS 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441): tablet d57077b2e3014da6a8052902dd6b283e (table test-table [id=110312da2937458ba807ffedeac6c6d9]) successfully deleted
I20260927 16:34:00.150615 8421 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Bootstrap starting.
I20260927 16:34:00.151584 8421 tablet_bootstrap.cc:654] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.151871 8420 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: No bootstrap required, opened a new log
I20260927 16:34:00.151948 8420 ts_tablet_manager.cc:1397] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 16:34:00.151968 8444 ts_tablet_manager.cc:1918] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.152056 8444 log.cc:1196] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/d57077b2e3014da6a8052902dd6b283e
I20260927 16:34:00.152369 8444 ts_tablet_manager.cc:1939] T d57077b2e3014da6a8052902dd6b283e P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting consensus metadata
I20260927 16:34:00.152482 8420 raft_consensus.cc:348] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.152594 8420 raft_consensus.cc:374] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.152626 8420 raft_consensus.cc:729] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.152745 8420 consensus_queue.cc:260] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.152992 8420 ts_tablet_manager.cc:1428] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.153199 8422 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Bootstrap starting.
I20260927 16:34:00.153705 7895 catalog_manager.cc:4841] TS 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735): tablet d57077b2e3014da6a8052902dd6b283e (table test-table [id=110312da2937458ba807ffedeac6c6d9]) successfully deleted
I20260927 16:34:00.154095 8422 tablet_bootstrap.cc:654] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.154943 8421 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: No bootstrap required, opened a new log
I20260927 16:34:00.155045 8421 ts_tablet_manager.cc:1397] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:34:00.155478 8421 raft_consensus.cc:348] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.155596 8421 raft_consensus.cc:374] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.155628 8421 raft_consensus.cc:729] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.155748 8421 consensus_queue.cc:260] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.155880 8422 tablet_bootstrap.cc:492] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: No bootstrap required, opened a new log
I20260927 16:34:00.155948 8422 ts_tablet_manager.cc:1397] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:00.156013 8421 ts_tablet_manager.cc:1428] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.156704 8422 raft_consensus.cc:348] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.156826 8422 raft_consensus.cc:374] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.156868 8422 raft_consensus.cc:729] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.156994 8422 consensus_queue.cc:260] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.157959 8422 ts_tablet_manager.cc:1428] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:34:00.192009 8427 raft_consensus.cc:482] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:00.192152 8427 raft_consensus.cc:504] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.193344 8427 leader_election.cc:288] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.198104 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.193685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44241 (local address 127.0.98.1:44735)
0927 16:34:00.193822 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.193827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.193842 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:00.194001 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.194005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.194062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.194147 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.194593 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.195277 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.196958 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.197595 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.197704 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.197770 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.197914 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.198008 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:34:00.198120 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.193540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44735 (local address 127.0.98.3:44241)
0927 16:34:00.193738 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.193754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.193851 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.194294 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.194297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.194317 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.194455 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.194464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.195407 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.195410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.196813 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.196823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.197707 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.197709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.197783 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.197802 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.197892 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.197968 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":97}
I20260927 16:34:00.198733 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc41fe5f1474fbb9f33b7fbfcb47732" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" is_pre_election: true
I20260927 16:34:00.199004 8066 raft_consensus.cc:2454] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:00.199120 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.193993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36789 (local address 127.0.98.2:43717)
0927 16:34:00.194128 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.194132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.194146 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:00.194224 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.194227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.194288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.194368 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.194757 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.195498 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.197827 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.198462 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.198559 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.198614 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.198927 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.198999 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:34:00.199564 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.193613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.3:36789)
0927 16:34:00.193976 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.193989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.194068 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.194495 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.194498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.194516 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.194633 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.194639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.195608 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.195611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.196832 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.196838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.198671 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.198675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.198792 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.198814 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.199319 (+ 505us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.199392 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":19,"thread_start_us":104,"threads_started":1}
I20260927 16:34:00.199559 8279 leader_election.cc:302] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, 176385af8648417a8336fd6d80ecf638; no voters:
I20260927 16:34:00.199924 8427 raft_consensus.cc:2781] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:00.200009 8427 raft_consensus.cc:482] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:00.200050 8427 raft_consensus.cc:3037] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.200138 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc41fe5f1474fbb9f33b7fbfcb47732" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" is_pre_election: true
I20260927 16:34:00.200277 8206 raft_consensus.cc:2454] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:00.201418 8427 raft_consensus.cc:504] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.201809 8427 leader_election.cc:288] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Requested vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.202207 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc41fe5f1474fbb9f33b7fbfcb47732" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
I20260927 16:34:00.202222 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc41fe5f1474fbb9f33b7fbfcb47732" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
I20260927 16:34:00.202380 8206 raft_consensus.cc:3037] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.202386 8066 raft_consensus.cc:3037] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.203343 8066 raft_consensus.cc:2454] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:00.203584 8206 raft_consensus.cc:2454] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:00.203652 8279 leader_election.cc:302] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, 176385af8648417a8336fd6d80ecf638; no voters:
I20260927 16:34:00.203827 8427 raft_consensus.cc:2781] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:00.204067 8427 raft_consensus.cc:686] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Becoming Leader. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Running, Role: LEADER
I20260927 16:34:00.204337 8427 consensus_queue.cc:237] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.206842 7907 catalog_manager.cc:5495] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176385af8648417a8336fd6d80ecf638 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "176385af8648417a8336fd6d80ecf638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } health_report { overall_health: HEALTHY } } }
Sep 27 16:34:00 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:34:00.261734 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.252005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43665 (local address 127.0.0.1:50412)
0927 16:34:00.252326 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.252344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.252466 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.252748 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.252750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.253122 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:00.253216 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.253222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.254266 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.254269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.254822 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.254829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.255437 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.255439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.255522 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.256203 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:00.256215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:00.257673 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:00.259844 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:00.259853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:00.259859 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:00.260103 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:00.260442 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:00.260444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:00.260451 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:00.260574 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:00.261063 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:00.261069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:00.261184 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.261283 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.261547 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260927 16:34:00.261861 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.252091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.98.62:43665)
0927 16:34:00.252243 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.252249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.252434 (+ 185us) server_negotiation.cc:407] Connection header received
0927 16:34:00.252560 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.252563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.252615 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.252720 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:00.253345 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.254144 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.254946 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.255319 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.255433 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.257840 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:00.257844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:00.257847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:00.257877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:00.259703 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:00.260216 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:00.260221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:00.260223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:00.260335 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:00.260686 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:00.260690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:00.260691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:00.260945 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:00.261060 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.261381 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.261657 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":58}
I20260927 16:34:00.266626 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:34:00.267525 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:34:00.272495 8046 tablet_service.cc:1467] Processing CreateTablet for tablet f43b9e7edc764ab589e596d17735ae2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315218b957874f71bb8c2b47a610253d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:34:00.272881 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43b9e7edc764ab589e596d17735ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.272991 8178 tablet_service.cc:1467] Processing CreateTablet for tablet f43b9e7edc764ab589e596d17735ae2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315218b957874f71bb8c2b47a610253d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:34:00.273370 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43b9e7edc764ab589e596d17735ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.275465 8421 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: Bootstrap starting.
I20260927 16:34:00.276371 8421 tablet_bootstrap.cc:654] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.276435 8422 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: Bootstrap starting.
I20260927 16:34:00.276798 8326 tablet_service.cc:1467] Processing CreateTablet for tablet f43b9e7edc764ab589e596d17735ae2c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=315218b957874f71bb8c2b47a610253d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:34:00.277148 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43b9e7edc764ab589e596d17735ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.277406 8422 tablet_bootstrap.cc:654] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.278555 8421 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: No bootstrap required, opened a new log
I20260927 16:34:00.278555 8422 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: No bootstrap required, opened a new log
I20260927 16:34:00.278637 8422 ts_tablet_manager.cc:1397] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:00.278637 8421 ts_tablet_manager.cc:1397] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:00.279115 8422 raft_consensus.cc:348] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.279224 8422 raft_consensus.cc:374] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.279258 8422 raft_consensus.cc:729] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.279340 8421 raft_consensus.cc:348] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.279383 8422 consensus_queue.cc:260] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.279434 8421 raft_consensus.cc:374] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.279479 8421 raft_consensus.cc:729] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.279590 8421 consensus_queue.cc:260] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.279652 8422 ts_tablet_manager.cc:1428] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.279857 8421 ts_tablet_manager.cc:1428] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 16:34:00.279956 8426 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.280077 8426 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.280236 8440 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.280357 8440 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.281859 8420 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: Bootstrap starting.
I20260927 16:34:00.282824 8420 tablet_bootstrap.cc:654] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.284158 8420 tablet_bootstrap.cc:492] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: No bootstrap required, opened a new log
I20260927 16:34:00.284260 8420 ts_tablet_manager.cc:1397] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:00.284696 8420 raft_consensus.cc:348] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.284824 8420 raft_consensus.cc:374] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.284859 8420 raft_consensus.cc:729] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.284981 8420 consensus_queue.cc:260] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.285349 8420 ts_tablet_manager.cc:1428] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.285534 8427 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.285657 8427 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.401574 8426 raft_consensus.cc:482] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:00.401713 8426 raft_consensus.cc:504] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.402791 8426 leader_election.cc:288] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441)
I20260927 16:34:00.406665 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.403156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42325 (local address 127.0.98.1:44735)
0927 16:34:00.403409 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.403415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.403435 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:34:00.403626 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.403631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.403683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.403771 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.404062 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.404568 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.405691 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.406078 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.406207 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.406271 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.406463 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.406533 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260927 16:34:00.406695 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.403034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44735 (local address 127.0.98.2:42325)
0927 16:34:00.403364 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.403379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.403492 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.403779 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.403782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.403797 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.403945 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.403950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.404675 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.404681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.405567 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.405576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.406218 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.406222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.406335 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.406353 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.406441 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.406506 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":6,"spinlock_wait_cycles":13056,"thread_start_us":119,"threads_started":1}
I20260927 16:34:00.406716 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.403166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47241 (local address 127.0.98.3:37441)
0927 16:34:00.403307 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.403312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.403505 (+ 193us) server_negotiation.cc:407] Connection header received
0927 16:34:00.403656 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.403661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.403709 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.403793 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:00.404077 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.404563 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.405703 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.406100 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.406255 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.406314 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.406510 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.406584 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:34:00.406785 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.403034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37441 (local address 127.0.98.2:47241)
0927 16:34:00.403398 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.403410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.403543 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.403804 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.403807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.403822 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:00.403966 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.403971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.404676 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.404680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.405577 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.405586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.406261 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.406265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.406359 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.406380 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.406493 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.406561 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":87,"thread_start_us":90,"threads_started":1}
I20260927 16:34:00.407120 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43b9e7edc764ab589e596d17735ae2c" candidate_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" is_pre_election: true
I20260927 16:34:00.407299 8066 raft_consensus.cc:2454] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88fadc568a043fc89e8b0a95b60e3a4 in term 0.
I20260927 16:34:00.407286 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43b9e7edc764ab589e596d17735ae2c" candidate_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176385af8648417a8336fd6d80ecf638" is_pre_election: true
I20260927 16:34:00.407409 8346 raft_consensus.cc:2454] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88fadc568a043fc89e8b0a95b60e3a4 in term 0.
I20260927 16:34:00.407708 8139 leader_election.cc:302] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.408026 8426 raft_consensus.cc:2781] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:00.408092 8426 raft_consensus.cc:482] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:00.408131 8426 raft_consensus.cc:3037] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.409195 8426 raft_consensus.cc:504] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.409513 8426 leader_election.cc:288] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441)
I20260927 16:34:00.409847 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43b9e7edc764ab589e596d17735ae2c" candidate_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
I20260927 16:34:00.409857 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43b9e7edc764ab589e596d17735ae2c" candidate_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176385af8648417a8336fd6d80ecf638"
I20260927 16:34:00.409986 8066 raft_consensus.cc:3037] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.410045 8346 raft_consensus.cc:3037] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.411005 8346 raft_consensus.cc:2454] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88fadc568a043fc89e8b0a95b60e3a4 in term 1.
I20260927 16:34:00.411194 8066 raft_consensus.cc:2454] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88fadc568a043fc89e8b0a95b60e3a4 in term 1.
I20260927 16:34:00.411305 8137 leader_election.cc:302] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176385af8648417a8336fd6d80ecf638, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.411541 8426 raft_consensus.cc:2781] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:00.411705 8426 raft_consensus.cc:686] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 LEADER]: Becoming Leader. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Running, Role: LEADER
I20260927 16:34:00.411958 8426 consensus_queue.cc:237] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.413138 8464 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: TxnStatusTablet state changed. Reason: New leader a88fadc568a043fc89e8b0a95b60e3a4. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.413277 8464 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:34:00.413565 8466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:34:00.413748 8466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:34:00.414541 7908 catalog_manager.cc:5495] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } health_report { overall_health: UNKNOWN } } }
I20260927 16:34:00.438378 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.434489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.0.98.2:43717)
0927 16:34:00.434622 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.434626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.434691 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:34:00.434895 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.434897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.434954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.435038 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.435286 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.435781 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.436968 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.437176 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.437250 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.437555 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.437961 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.438044 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.438158 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.438250 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:34:00.438397 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.434389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:41872)
0927 16:34:00.434628 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.434643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.434755 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.435074 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.435077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.435086 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:34:00.435199 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.435205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.435878 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.435881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.436851 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.436861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.437280 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.437284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.437372 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.437422 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.438053 (+ 631us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.438068 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.438144 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.438255 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 16:34:00.441184 8346 raft_consensus.cc:1261] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Refusing update from remote peer a88fadc568a043fc89e8b0a95b60e3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.441342 8066 raft_consensus.cc:1261] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Refusing update from remote peer a88fadc568a043fc89e8b0a95b60e3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.441711 8464 consensus_queue.cc:1035] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.441931 8426 consensus_queue.cc:1035] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.443375 8427 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: TxnStatusTablet state changed. Reason: New leader a88fadc568a043fc89e8b0a95b60e3a4. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.443490 8427 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.443629 8440 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: TxnStatusTablet state changed. Reason: New leader a88fadc568a043fc89e8b0a95b60e3a4. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.443754 8440 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.445587 8426 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.445693 8426 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:34:00.445654 8471 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.445748 8471 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P a88fadc568a043fc89e8b0a95b60e3a4: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:34:00.445776 8435 mvcc.cc:205] Tried to move back new op lower bound from 7333997938442137600 to 7333997938329657344. Current Snapshot: MvccSnapshot[applied={T|T < 7333997938442137600}]
I20260927 16:34:00.446866 8434 mvcc.cc:205] Tried to move back new op lower bound from 7333997938442137600 to 7333997938329657344. Current Snapshot: MvccSnapshot[applied={T|T < 7333997938442137600}]
I20260927 16:34:00.447033 8440 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.447118 8440 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 09af8e633b2e46e9a5409a7e9c5338fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.447590 8427 tablet_replica.cc:439] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } } }
I20260927 16:34:00.447691 8427 tablet_replica.cc:442] T f43b9e7edc764ab589e596d17735ae2c P 176385af8648417a8336fd6d80ecf638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:34:00.447760 8436 mvcc.cc:205] Tried to move back new op lower bound from 7333997938442137600 to 7333997938329657344. Current Snapshot: MvccSnapshot[applied={T|T < 7333997938442137600 or (T in {7333997938442137600})}]
I20260927 16:34:00.454384 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.450888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.0.98.3:37441)
0927 16:34:00.451024 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.451028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.451040 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:34:00.451172 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.451175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.451234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.451339 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.451571 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.452078 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.452911 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.453153 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.453260 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.453534 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.453964 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.454094 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.454155 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.454230 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:34:00.466212 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.461828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40851 (local address 127.0.98.2:43717)
0927 16:34:00.462003 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.462008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.462262 (+ 254us) server_negotiation.cc:407] Connection header received
0927 16:34:00.462519 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.462522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.462567 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.462625 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.463342 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.463845 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.465115 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.465383 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.465471 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.465775 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.465843 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.465895 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.466015 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.466057 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 16:34:00.466265 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.461745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.3:40851)
0927 16:34:00.462135 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.462152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.462351 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.462665 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.462670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.462682 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:34:00.463230 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.463238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.463940 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.463944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.464993 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.465003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.465491 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.465494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.465593 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.465648 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.465921 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.465926 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.465999 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.466079 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":157,"threads_started":1}
I20260927 16:34:00.473280 8066 raft_consensus.cc:1261] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.473277 8206 raft_consensus.cc:1261] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.473773 8451 consensus_queue.cc:1035] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.474013 8427 consensus_queue.cc:1035] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.497696 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.494022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44749 (local address 127.0.98.3:37441)
0927 16:34:00.494134 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.494138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.494152 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:00.494264 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.494266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.494317 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.494381 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.494797 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.495337 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.496391 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.496713 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.496826 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.497093 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.497175 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.497248 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.497490 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.497548 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:34:00.497727 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.493680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37441 (local address 127.0.98.2:44749)
0927 16:34:00.494032 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:00.494048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:00.494145 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:00.494509 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:00.494512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:00.494521 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:34:00.494664 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.494671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.495463 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.495466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.496100 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:00.496108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.496824 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.496828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:00.496922 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.496975 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.497268 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.497275 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:34:00.497457 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 16:34:00.497561 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260927 16:34:00.524305 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.524490 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.527101 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Sending DeleteTablet for 3 replicas of tablet 1cc41fe5f1474fbb9f33b7fbfcb47732
I20260927 16:34:00.527874 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260927 16:34:00.527930 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 16:34:00.528069 8444 tablet_replica.cc:330] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: stopping tablet replica
I20260927 16:34:00.528127 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260927 16:34:00.528185 8444 raft_consensus.cc:2229] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.528292 8445 tablet_replica.cc:330] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: stopping tablet replica
I20260927 16:34:00.528302 8444 raft_consensus.cc:2258] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.528393 8445 raft_consensus.cc:2229] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.528497 8445 raft_consensus.cc:2258] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.528649 8446 tablet_replica.cc:330] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: stopping tablet replica
I20260927 16:34:00.528754 8446 raft_consensus.cc:2229] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:34:00.529048 8446 raft_consensus.cc:2258] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.529850 8444 ts_tablet_manager.cc:1905] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.530001 8446 ts_tablet_manager.cc:1905] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.532887 8446 ts_tablet_manager.cc:1918] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:34:00.532979 8446 log.cc:1196] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/1cc41fe5f1474fbb9f33b7fbfcb47732
I20260927 16:34:00.533227 8446 ts_tablet_manager.cc:1939] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 176385af8648417a8336fd6d80ecf638: Deleting consensus metadata
I20260927 16:34:00.533913 7893 catalog_manager.cc:4841] TS 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441): tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (table test-table [id=bfba26459cc040f7af39cbe2ac772dad]) successfully deleted
I20260927 16:34:00.534732 8445 ts_tablet_manager.cc:1905] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.536819 8444 ts_tablet_manager.cc:1918] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:34:00.536983 8444 log.cc:1196] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/1cc41fe5f1474fbb9f33b7fbfcb47732
I20260927 16:34:00.537307 8444 ts_tablet_manager.cc:1939] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting consensus metadata
I20260927 16:34:00.537669 8445 ts_tablet_manager.cc:1918] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:34:00.537758 8445 log.cc:1196] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/1cc41fe5f1474fbb9f33b7fbfcb47732
I20260927 16:34:00.538039 8445 ts_tablet_manager.cc:1939] T 1cc41fe5f1474fbb9f33b7fbfcb47732 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting consensus metadata
I20260927 16:34:00.538244 7895 catalog_manager.cc:4841] TS 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735): tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (table test-table [id=bfba26459cc040f7af39cbe2ac772dad]) successfully deleted
I20260927 16:34:00.538756 7893 catalog_manager.cc:4841] TS a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717): tablet 1cc41fe5f1474fbb9f33b7fbfcb47732 (table test-table [id=bfba26459cc040f7af39cbe2ac772dad]) successfully deleted
Sep 27 16:34:00 dist-test-slave-t27c krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:34:00 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:00 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:34:00.570972 7958 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:00.562891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.98.62:43665)
0927 16:34:00.563019 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.563023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.563288 (+ 265us) server_negotiation.cc:407] Connection header received
0927 16:34:00.563411 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.563415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.563474 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.563572 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:00.564551 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.565401 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.566654 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.567012 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.567128 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.570827 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50416: BlockingRecv error: recv got EOF from 127.0.0.1:50416 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 16:34:00 dist-test-slave-t27c krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:34:00.583185 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.572358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.0.98.62:43665)
0927 16:34:00.572462 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.572466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.573211 (+ 745us) server_negotiation.cc:407] Connection header received
0927 16:34:00.573315 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.573318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.573363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.573445 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:00.574131 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.574913 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.576036 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.576362 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.576462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.579458 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:00.579465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:00.579468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:00.579504 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:00.581291 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:00.581786 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:00.581790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:00.581792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:00.581885 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:00.582227 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:00.582245 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:00.582247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:00.582494 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:00.582631 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.582883 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.583010 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:34:00.586449 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:34:00.586944 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:34:00.592381 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 (DEFAULT_TABLE table=test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.592442 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 (DEFAULT_TABLE table=test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.592669 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ff6231b2d24ae9b91c448ac384b330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.592720 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ff6231b2d24ae9b91c448ac384b330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.594475 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 (DEFAULT_TABLE table=test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.594739 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ff6231b2d24ae9b91c448ac384b330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.595253 8421 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Bootstrap starting.
I20260927 16:34:00.596102 8421 tablet_bootstrap.cc:654] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.597054 8422 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Bootstrap starting.
I20260927 16:34:00.597714 8421 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: No bootstrap required, opened a new log
I20260927 16:34:00.597790 8421 ts_tablet_manager.cc:1397] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:00.597829 8422 tablet_bootstrap.cc:654] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.598204 8421 raft_consensus.cc:348] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.598310 8421 raft_consensus.cc:374] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.598338 8421 raft_consensus.cc:729] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.598459 8421 consensus_queue.cc:260] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.598692 8421 ts_tablet_manager.cc:1428] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.600322 8420 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Bootstrap starting.
I20260927 16:34:00.601280 8420 tablet_bootstrap.cc:654] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.602757 8422 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: No bootstrap required, opened a new log
I20260927 16:34:00.602758 8420 tablet_bootstrap.cc:492] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: No bootstrap required, opened a new log
I20260927 16:34:00.602861 8420 ts_tablet_manager.cc:1397] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:00.602869 8422 ts_tablet_manager.cc:1397] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 16:34:00.603225 8422 raft_consensus.cc:348] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.603348 8422 raft_consensus.cc:374] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.603292 8420 raft_consensus.cc:348] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.603389 8422 raft_consensus.cc:729] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.603392 8420 raft_consensus.cc:374] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.603435 8420 raft_consensus.cc:729] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.603524 8422 consensus_queue.cc:260] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.603557 8420 consensus_queue.cc:260] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.603816 8420 ts_tablet_manager.cc:1428] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.603827 8422 ts_tablet_manager.cc:1428] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.695154 8477 raft_consensus.cc:482] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:00.695324 8477 raft_consensus.cc:504] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.695828 8477 leader_election.cc:288] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.696295 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ff6231b2d24ae9b91c448ac384b330" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" is_pre_election: true
I20260927 16:34:00.696297 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ff6231b2d24ae9b91c448ac384b330" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" is_pre_election: true
I20260927 16:34:00.696475 8206 raft_consensus.cc:2454] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:00.696471 8066 raft_consensus.cc:2454] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:00.696799 8277 leader_election.cc:302] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176385af8648417a8336fd6d80ecf638, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.697012 8477 raft_consensus.cc:2781] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:00.697077 8477 raft_consensus.cc:482] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:00.697117 8477 raft_consensus.cc:3037] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.698161 8477 raft_consensus.cc:504] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.698738 8477 leader_election.cc:288] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Requested vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:00.699167 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ff6231b2d24ae9b91c448ac384b330" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
I20260927 16:34:00.699165 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ff6231b2d24ae9b91c448ac384b330" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
I20260927 16:34:00.699317 8206 raft_consensus.cc:3037] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.699322 8066 raft_consensus.cc:3037] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:00.700551 8206 raft_consensus.cc:2454] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:00.700703 8066 raft_consensus.cc:2454] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:00.700865 8277 leader_election.cc:302] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176385af8648417a8336fd6d80ecf638, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:00.701020 8477 raft_consensus.cc:2781] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:00.701140 8477 raft_consensus.cc:686] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Becoming Leader. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Running, Role: LEADER
I20260927 16:34:00.701301 8477 consensus_queue.cc:237] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.703224 7909 catalog_manager.cc:5495] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176385af8648417a8336fd6d80ecf638 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "176385af8648417a8336fd6d80ecf638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } health_report { overall_health: HEALTHY } } }
I20260927 16:34:00.731518 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:00.728213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.98.3:37441)
0927 16:34:00.728363 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:00.728368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:00.728382 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:00.728514 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:00.728521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:00.728566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:00.728647 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:00.728890 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.729526 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.730454 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:00.730679 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:00.730787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:00.731053 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:00.731129 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:00.731208 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:00.731308 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 16:34:00.731361 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:34:00.733676 8206 raft_consensus.cc:1261] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.733773 8066 raft_consensus.cc:1261] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:00.734125 8477 consensus_queue.cc:1035] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.734287 8451 consensus_queue.cc:1035] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:00.741495 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.741654 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:00.743634 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Sending DeleteTablet for 3 replicas of tablet 57ff6231b2d24ae9b91c448ac384b330
I20260927 16:34:00.744184 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260927 16:34:00.744218 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 16:34:00.744323 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ff6231b2d24ae9b91c448ac384b330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260927 16:34:00.744377 8444 tablet_replica.cc:330] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: stopping tablet replica
I20260927 16:34:00.744385 8445 tablet_replica.cc:330] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: stopping tablet replica
I20260927 16:34:00.744498 8445 raft_consensus.cc:2229] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.744503 8444 raft_consensus.cc:2229] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:00.744537 8446 tablet_replica.cc:330] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: stopping tablet replica
I20260927 16:34:00.744628 8445 raft_consensus.cc:2258] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.744638 8444 raft_consensus.cc:2258] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.744637 8446 raft_consensus.cc:2229] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:34:00.744808 8446 raft_consensus.cc:2258] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:00.745201 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:34:00.745402 8445 ts_tablet_manager.cc:1905] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.745533 8444 ts_tablet_manager.cc:1905] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:34:00.745664 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:34:00.747941 8444 ts_tablet_manager.cc:1918] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.747961 8445 ts_tablet_manager.cc:1918] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.748031 8445 log.cc:1196] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/57ff6231b2d24ae9b91c448ac384b330
I20260927 16:34:00.748031 8444 log.cc:1196] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/57ff6231b2d24ae9b91c448ac384b330
I20260927 16:34:00.748292 8444 ts_tablet_manager.cc:1939] T 57ff6231b2d24ae9b91c448ac384b330 P 09af8e633b2e46e9a5409a7e9c5338fc: Deleting consensus metadata
I20260927 16:34:00.748306 8445 ts_tablet_manager.cc:1939] T 57ff6231b2d24ae9b91c448ac384b330 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting consensus metadata
I20260927 16:34:00.748941 8446 ts_tablet_manager.cc:1905] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:00.749226 7895 catalog_manager.cc:4841] TS 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735): tablet 57ff6231b2d24ae9b91c448ac384b330 (table test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]) successfully deleted
I20260927 16:34:00.749755 7893 catalog_manager.cc:4841] TS a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717): tablet 57ff6231b2d24ae9b91c448ac384b330 (table test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]) successfully deleted
I20260927 16:34:00.751379 8446 ts_tablet_manager.cc:1918] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:00.751458 8446 log.cc:1196] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/57ff6231b2d24ae9b91c448ac384b330
I20260927 16:34:00.751497 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 78cfccf587ed447aba9ce32901f82e50 (DEFAULT_TABLE table=test-table [id=70b17df46b254f5cac28788fc598a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.751562 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 78cfccf587ed447aba9ce32901f82e50 (DEFAULT_TABLE table=test-table [id=70b17df46b254f5cac28788fc598a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.751713 8446 ts_tablet_manager.cc:1939] T 57ff6231b2d24ae9b91c448ac384b330 P 176385af8648417a8336fd6d80ecf638: Deleting consensus metadata
I20260927 16:34:00.751765 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cfccf587ed447aba9ce32901f82e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.751848 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cfccf587ed447aba9ce32901f82e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.752405 7893 catalog_manager.cc:4841] TS 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441): tablet 57ff6231b2d24ae9b91c448ac384b330 (table test-table [id=c87ed1a92fb6414ba29cfccfe1413f41]) successfully deleted
I20260927 16:34:00.752444 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 78cfccf587ed447aba9ce32901f82e50 (DEFAULT_TABLE table=test-table [id=70b17df46b254f5cac28788fc598a1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:00.752669 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cfccf587ed447aba9ce32901f82e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:00.753684 8420 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Bootstrap starting.
I20260927 16:34:00.753854 8422 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Bootstrap starting.
I20260927 16:34:00.754540 8420 tablet_bootstrap.cc:654] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.754559 8422 tablet_bootstrap.cc:654] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.754729 8421 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: Bootstrap starting.
I20260927 16:34:00.755471 8421 tablet_bootstrap.cc:654] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:00.756356 8421 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: No bootstrap required, opened a new log
I20260927 16:34:00.756484 8421 ts_tablet_manager.cc:1397] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:34:00.756764 8421 raft_consensus.cc:348] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.756840 8421 raft_consensus.cc:374] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.756860 8421 raft_consensus.cc:729] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af8e633b2e46e9a5409a7e9c5338fc, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.756984 8421 consensus_queue.cc:260] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.757265 8421 ts_tablet_manager.cc:1428] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.758096 8420 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: No bootstrap required, opened a new log
I20260927 16:34:00.758140 8422 tablet_bootstrap.cc:492] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: No bootstrap required, opened a new log
I20260927 16:34:00.758246 8422 ts_tablet_manager.cc:1397] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:34:00.758271 8420 ts_tablet_manager.cc:1397] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 16:34:00.758617 8420 raft_consensus.cc:348] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.758701 8420 raft_consensus.cc:374] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.758730 8420 raft_consensus.cc:729] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.758656 8422 raft_consensus.cc:348] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.758770 8422 raft_consensus.cc:374] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:00.758806 8422 raft_consensus.cc:729] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88fadc568a043fc89e8b0a95b60e3a4, State: Initialized, Role: FOLLOWER
I20260927 16:34:00.758833 8420 consensus_queue.cc:260] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.758913 8422 consensus_queue.cc:260] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:00.759040 8420 ts_tablet_manager.cc:1428] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:00.759183 8422 ts_tablet_manager.cc:1428] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:34:01.069228 8477 raft_consensus.cc:482] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:01.069370 8477 raft_consensus.cc:504] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:01.069871 8477 leader_election.cc:288] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:01.070331 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cfccf587ed447aba9ce32901f82e50" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" is_pre_election: true
I20260927 16:34:01.070358 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cfccf587ed447aba9ce32901f82e50" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" is_pre_election: true
I20260927 16:34:01.070506 8066 raft_consensus.cc:2454] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:01.070521 8206 raft_consensus.cc:2454] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 0.
I20260927 16:34:01.070895 8279 leader_election.cc:302] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09af8e633b2e46e9a5409a7e9c5338fc, 176385af8648417a8336fd6d80ecf638; no voters:
I20260927 16:34:01.071126 8477 raft_consensus.cc:2781] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:01.071189 8477 raft_consensus.cc:482] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:01.071230 8477 raft_consensus.cc:3037] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:01.072330 8477 raft_consensus.cc:504] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:01.072685 8477 leader_election.cc:288] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Requested vote from peers 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735), a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717)
I20260927 16:34:01.073029 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cfccf587ed447aba9ce32901f82e50" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af8e633b2e46e9a5409a7e9c5338fc"
I20260927 16:34:01.073046 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cfccf587ed447aba9ce32901f82e50" candidate_uuid: "176385af8648417a8336fd6d80ecf638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88fadc568a043fc89e8b0a95b60e3a4"
I20260927 16:34:01.073167 8206 raft_consensus.cc:3037] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:01.073174 8066 raft_consensus.cc:3037] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:01.074021 8206 raft_consensus.cc:2454] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:01.074117 8066 raft_consensus.cc:2454] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176385af8648417a8336fd6d80ecf638 in term 1.
I20260927 16:34:01.074316 8277 leader_election.cc:302] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176385af8648417a8336fd6d80ecf638, a88fadc568a043fc89e8b0a95b60e3a4; no voters:
I20260927 16:34:01.074517 8477 raft_consensus.cc:2781] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:01.074618 8477 raft_consensus.cc:686] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Becoming Leader. State: Replica: 176385af8648417a8336fd6d80ecf638, State: Running, Role: LEADER
I20260927 16:34:01.074771 8477 consensus_queue.cc:237] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } }
I20260927 16:34:01.076356 7909 catalog_manager.cc:5495] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176385af8648417a8336fd6d80ecf638 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "176385af8648417a8336fd6d80ecf638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176385af8648417a8336fd6d80ecf638" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37441 } health_report { overall_health: HEALTHY } } }
I20260927 16:34:01.130796 8066 raft_consensus.cc:1261] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:01.130834 8206 raft_consensus.cc:1261] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Refusing update from remote peer 176385af8648417a8336fd6d80ecf638: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:01.131247 8451 consensus_queue.cc:1035] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88fadc568a043fc89e8b0a95b60e3a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:01.131407 8477 consensus_queue.cc:1035] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09af8e633b2e46e9a5409a7e9c5338fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:01.163529 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:01.163678 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:01.165632 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97627fd83775479b8f5701ee01d860ea: Sending DeleteTablet for 3 replicas of tablet 78cfccf587ed447aba9ce32901f82e50
I20260927 16:34:01.166224 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cfccf587ed447aba9ce32901f82e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260927 16:34:01.166325 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cfccf587ed447aba9ce32901f82e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 16:34:01.166436 8444 tablet_replica.cc:330] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc: stopping tablet replica
I20260927 16:34:01.166543 8445 tablet_replica.cc:330] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: stopping tablet replica
I20260927 16:34:01.166575 8444 raft_consensus.cc:2229] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:01.166680 8445 raft_consensus.cc:2229] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:01.166697 8444 raft_consensus.cc:2258] T 78cfccf587ed447aba9ce32901f82e50 P 09af8e633b2e46e9a5409a7e9c5338fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:01.166802 8445 raft_consensus.cc:2258] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:01.166996 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cfccf587ed447aba9ce32901f82e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260927 16:34:01.167130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7981
I20260927 16:34:01.167268 8446 tablet_replica.cc:330] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: stopping tablet replica
I20260927 16:34:01.167410 8446 raft_consensus.cc:2229] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:34:01.167598 8446 raft_consensus.cc:2258] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:01.169272 8446 ts_tablet_manager.cc:1905] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:01.169677 8445 ts_tablet_manager.cc:1905] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:01.171422 8446 ts_tablet_manager.cc:1918] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:34:01.171518 8446 log.cc:1196] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/78cfccf587ed447aba9ce32901f82e50
I20260927 16:34:01.171749 8446 ts_tablet_manager.cc:1939] T 78cfccf587ed447aba9ce32901f82e50 P 176385af8648417a8336fd6d80ecf638: Deleting consensus metadata
I20260927 16:34:01.172418 7893 catalog_manager.cc:4841] TS 176385af8648417a8336fd6d80ecf638 (127.0.98.3:37441): tablet 78cfccf587ed447aba9ce32901f82e50 (table test-table [id=70b17df46b254f5cac28788fc598a1c7]) successfully deleted
I20260927 16:34:01.172652 8445 ts_tablet_manager.cc:1918] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:34:01.172739 8445 log.cc:1196] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/78cfccf587ed447aba9ce32901f82e50
I20260927 16:34:01.173000 8445 ts_tablet_manager.cc:1939] T 78cfccf587ed447aba9ce32901f82e50 P a88fadc568a043fc89e8b0a95b60e3a4: Deleting consensus metadata
I20260927 16:34:01.173702 7893 catalog_manager.cc:4841] TS a88fadc568a043fc89e8b0a95b60e3a4 (127.0.98.2:43717): tablet 78cfccf587ed447aba9ce32901f82e50 (table test-table [id=70b17df46b254f5cac28788fc598a1c7]) successfully deleted
W20260927 16:34:01.176645 7895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44735 (error 108)
I20260927 16:34:01.176821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8121
I20260927 16:34:01.177897 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:01.177486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44735 (local address 127.0.0.1:56914)
0927 16:34:01.177746 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:01.177807 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":58,"threads_started":1}
W20260927 16:34:01.178154 7895 catalog_manager.cc:4584] TS 09af8e633b2e46e9a5409a7e9c5338fc (127.0.98.1:44735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78cfccf587ed447aba9ce32901f82e50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44735: connect: Connection refused (error 111)
I20260927 16:34:01.184865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8261
I20260927 16:34:01.191565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 7877
2026-09-27T16:34:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3558 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:34:01 dist-test-slave-t27c krb5kdc[8515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:34:01 dist-test-slave-t27c krb5kdc[8515](info): set up 2 sockets
Sep 27 16:34:01 dist-test-slave-t27c krb5kdc[8515](info): commencing operation
krb5kdc: starting...
W20260927 16:34:02.243021 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:34:02 dist-test-slave-t27c krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:34:02.365880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:02.476563 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:02.476897 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:02.476976 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:02.481046 8528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:02.481122 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:02.481160 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:02.481196 8528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:02.481230 8528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:02.486567 8528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:02.487637 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:02.488691 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:02.497361 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:02.497445 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:02.497536 8528 server_base.cc:1034] running on GCE node
W20260927 16:34:02.497382 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:02.498247 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:02.499352 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:02.500516 8528 hybrid_clock.cc:648] HybridClock initialized: now 1790526842500485 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T16:34:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1320 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:34:02 dist-test-slave-t27c krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:34:02 dist-test-slave-t27c krb5kdc[8541](info): set up 2 sockets
Sep 27 16:34:02 dist-test-slave-t27c krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260927 16:34:03.599244 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:34:03 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:34:03.722517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:03.834729 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:03.835014 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:03.835057 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:03.839221 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:03.839284 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:03.839304 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:03.839321 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:03.839336 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:03.844584 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:03.845598 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:03.846585 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:03.855315 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:03.855338 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:03.855357 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:03.855561 8554 server_base.cc:1034] running on GCE node
I20260927 16:34:03.856171 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:03.857321 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:03.858503 8554 hybrid_clock.cc:648] HybridClock initialized: now 1790526843858467 us; error 48 us; skew 500 ppm
Sep 27 16:34:03 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:03.861471 8554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:34:03.862627 8554 webserver.cc:460] Webserver started at http://127.0.98.62:46701/ using document root <none> and password file <none>
I20260927 16:34:03.863245 8554 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:03.863343 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:03.863602 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:03.865819 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "8437510206fb474e8427c392f16bc021"
format_stamp: "Formatted at 2026-09-27 16:34:03 on dist-test-slave-t27c"
server_key: "07c6e0149f19a97061161f5dfe248a03"
server_key_iv: "71166958a2a0e053de8747ef383a29ea"
server_key_version: "encryptionkey@0"
I20260927 16:34:03.866446 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "8437510206fb474e8427c392f16bc021"
format_stamp: "Formatted at 2026-09-27 16:34:03 on dist-test-slave-t27c"
server_key: "07c6e0149f19a97061161f5dfe248a03"
server_key_iv: "71166958a2a0e053de8747ef383a29ea"
server_key_version: "encryptionkey@0"
I20260927 16:34:03.870364 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:34:03.872939 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:03.874261 8554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:03.874395 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8437510206fb474e8427c392f16bc021"
format_stamp: "Formatted at 2026-09-27 16:34:03 on dist-test-slave-t27c"
server_key: "07c6e0149f19a97061161f5dfe248a03"
server_key_iv: "71166958a2a0e053de8747ef383a29ea"
server_key_version: "encryptionkey@0"
I20260927 16:34:03.874507 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:03.897148 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:03.900552 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:03.900771 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:03.908797 8554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38445
I20260927 16:34:03.908819 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38445 every 8 connection(s)
I20260927 16:34:03.909857 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:34:03.912619 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:03.918088 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Bootstrap starting.
I20260927 16:34:03.918654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8554
I20260927 16:34:03.918780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:34:03.919054 392 external_mini_cluster.cc:1428] Setting key 2decca3eb533835a4b3c3577d40ea029
I20260927 16:34:03.920830 8622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:03.921711 8622 log.cc:824] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:03.923882 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: No bootstrap required, opened a new log
I20260927 16:34:03.927770 8622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:03.928012 8622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:03.928097 8622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8437510206fb474e8427c392f16bc021, State: Initialized, Role: FOLLOWER
Sep 27 16:34:03 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:03.928539 8622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:03.928721 8622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:34:03.928799 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:34:03.928944 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:03.930217 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:03.930603 8622 leader_election.cc:302] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8437510206fb474e8427c392f16bc021; no voters:
I20260927 16:34:03.930926 8622 leader_election.cc:288] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:34:03.931026 8627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:03.931254 8627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 LEADER]: Becoming Leader. State: Replica: 8437510206fb474e8427c392f16bc021, State: Running, Role: LEADER
I20260927 16:34:03.931602 8627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:03.932302 8622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:34:03.933171 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8437510206fb474e8427c392f16bc021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } } }
I20260927 16:34:03.933283 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:03.933660 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8437510206fb474e8427c392f16bc021. Latest consensus state: current_term: 1 leader_uuid: "8437510206fb474e8427c392f16bc021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } } }
I20260927 16:34:03.933815 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:03.934114 8636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:34:03.935303 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:03.920394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.98.62:38445)
0927 16:34:03.920838 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:03.920858 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:03.920902 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:34:03.921755 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:03.921779 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:03.922275 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:03.922617 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:03.923261 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:03.924721 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:03.925559 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:03.926300 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:03.926487 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:03.929160 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:03.929170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:03.929188 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:03.929219 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:03.931265 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:03.931784 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:03.931790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:03.931795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:03.931928 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:03.932240 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:03.932243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:03.932244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:03.932758 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:03.933696 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:03.933854 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:34:03.934209 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":110,"threads_started":1}
I20260927 16:34:03.938432 8636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:34:03.944024 8636 catalog_manager.cc:1347] Generated new cluster ID: 30ab75fe9c1448f48884ce9d8d7620da
I20260927 16:34:03.944121 8636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:34:03.955024 8636 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:34:03.955987 8636 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:34:03.962574 8636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Generated new TSK 0
I20260927 16:34:03.963248 8636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:34:03.996515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:04.120368 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:04.120633 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:04.120676 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:04.125438 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:04.125512 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:04.125620 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:34:04.130986 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:04.132054 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:04.133196 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:04.143029 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.143020 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.143029 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.143316 8648 server_base.cc:1034] running on GCE node
I20260927 16:34:04.143842 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:04.144565 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:04.145772 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790526844145750 us; error 45 us; skew 500 ppm
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:04.148597 8648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:34:04.149559 8648 webserver.cc:460] Webserver started at http://127.0.98.1:37171/ using document root <none> and password file <none>
I20260927 16:34:04.150377 8648 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:04.150487 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:04.150740 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:04.153354 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "06eefd57b7cf4c4891ccf154f9fe8a37"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "e6f1c32d2740f1036f1baac3c5332681"
server_key_iv: "987e8b80263e1cb91fbf664bf4988508"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.154076 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06eefd57b7cf4c4891ccf154f9fe8a37"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "e6f1c32d2740f1036f1baac3c5332681"
server_key_iv: "987e8b80263e1cb91fbf664bf4988508"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.157832 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:34:04.160557 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.162102 8648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:04.162249 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "06eefd57b7cf4c4891ccf154f9fe8a37"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "e6f1c32d2740f1036f1baac3c5332681"
server_key_iv: "987e8b80263e1cb91fbf664bf4988508"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.162381 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:04.180941 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:04.184767 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:04.184983 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:04.185647 8648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:04.186559 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:04.186609 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.186648 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:04.186699 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.195433 8648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260927 16:34:04.195456 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260927 16:34:04.196420 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:34:04.203886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8648
I20260927 16:34:04.204051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:34:04.204322 392 external_mini_cluster.cc:1428] Setting key ccdbe9070d6adb29453180e9ef190cab
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:04.210824 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.198344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36269 (local address 127.0.98.62:38445)
0927 16:34:04.198490 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:04.198494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:04.199389 (+ 895us) server_negotiation.cc:407] Connection header received
0927 16:34:04.200796 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:04.200799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:04.200857 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:04.200978 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:04.202125 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.202715 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.203861 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.204197 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.204306 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.207251 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:04.207261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:04.207264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:04.207286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:04.208756 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.209257 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.209262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.209264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.209339 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.209682 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.209687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.209689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.209864 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:04.209970 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:04.210529 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 16:34:04.210657 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:34:04.211624 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.198788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:36269)
0927 16:34:04.199240 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:04.199282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:04.200515 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:04.201152 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:04.201159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:04.201504 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:04.201984 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.201996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.202843 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.202852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.203728 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.203737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.204346 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.204350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.204516 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.205203 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:04.205223 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:04.207088 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:04.208876 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.208883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.208898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.209155 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.209435 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.209438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.209440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.209586 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.209975 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:04.209983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:04.210297 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 16:34:04.210488 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 16:34:04.210804 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260927 16:34:04.212838 8777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38445
I20260927 16:34:04.213104 8777 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:34:04.213708 8777 heartbeater.cc:511] Master 127.0.98.62:38445 requested a full tablet report, sending...
I20260927 16:34:04.215257 8586 ts_manager.cc:194] Registered new tserver with Master: 06eefd57b7cf4c4891ccf154f9fe8a37 (127.0.98.1:33999)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:34:04.216529 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:34:04.233297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:04.344177 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:04.344475 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:04.344519 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:04.348788 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:04.348850 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:04.348943 8782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:34:04.354458 8782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:04.355572 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:04.356622 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:04.366191 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.366338 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.366232 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.366423 8782 server_base.cc:1034] running on GCE node
I20260927 16:34:04.366900 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:04.367551 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:04.368737 8782 hybrid_clock.cc:648] HybridClock initialized: now 1790526844368723 us; error 28 us; skew 500 ppm
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:04.371430 8782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:34:04.372313 8782 webserver.cc:460] Webserver started at http://127.0.98.2:41967/ using document root <none> and password file <none>
I20260927 16:34:04.372888 8782 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:04.372980 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:04.373210 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:04.375384 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "11822b15868948a288499d9589d21d87"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "4be031bd3eea6a0047a7d1e4185dbdb6"
server_key_iv: "8e734a9c6892609025b8fbff32e38c9e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.375952 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11822b15868948a288499d9589d21d87"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "4be031bd3eea6a0047a7d1e4185dbdb6"
server_key_iv: "8e734a9c6892609025b8fbff32e38c9e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.379760 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:34:04.382345 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.383631 8782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:04.383776 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "11822b15868948a288499d9589d21d87"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "4be031bd3eea6a0047a7d1e4185dbdb6"
server_key_iv: "8e734a9c6892609025b8fbff32e38c9e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.383904 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:04.402611 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:04.405543 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:04.405762 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:04.406440 8782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:04.407287 8782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:04.407357 8782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.407428 8782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:04.407478 8782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.416253 8782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38733
I20260927 16:34:04.416272 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38733 every 8 connection(s)
I20260927 16:34:04.417200 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:34:04.419951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8782
I20260927 16:34:04.420095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:34:04.420413 392 external_mini_cluster.cc:1428] Setting key 61ca1b9714c0402a6d8dfbce3277979c
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:34:04.433314 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.419062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42573 (local address 127.0.98.62:38445)
0927 16:34:04.419209 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:04.419212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:04.420239 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 16:34:04.421999 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:04.422002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:04.422052 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:04.422165 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:04.423636 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.424218 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.426137 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.426428 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.426513 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.429544 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:04.429551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:04.429557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:04.429594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:04.431183 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.431686 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.431689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.431691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.431777 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.432153 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.432156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.432158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.432352 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:04.432504 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:04.432992 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 16:34:04.433126 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:34:04.435490 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.419477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:42573)
0927 16:34:04.420018 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:04.420089 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:04.421740 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:04.422381 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:04.422390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:04.422867 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:04.423425 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.423441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.424362 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.424367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.425972 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.425988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.426598 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.426603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.426855 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.427550 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:04.427578 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:04.429355 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:04.431286 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.431291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.431305 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.431564 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.431874 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.431878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.431880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.432024 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.432469 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:04.432477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:04.432752 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 16:34:04.432954 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 16:34:04.433379 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260927 16:34:04.436726 8911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38445
I20260927 16:34:04.437001 8911 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:04.437631 8911 heartbeater.cc:511] Master 127.0.98.62:38445 requested a full tablet report, sending...
I20260927 16:34:04.438882 8586 ts_manager.cc:194] Registered new tserver with Master: 11822b15868948a288499d9589d21d87 (127.0.98.2:38733)
I20260927 16:34:04.439694 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:34:04.449246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:04.559054 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:04.559312 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:04.559355 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:04.563519 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:04.563572 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:04.563632 8916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:34:04.568814 8916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:04.569728 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:04.570713 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:04.580245 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.580284 8916 server_base.cc:1034] running on GCE node
W20260927 16:34:04.580245 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.580245 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.580839 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:04.581547 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:04.582715 8916 hybrid_clock.cc:648] HybridClock initialized: now 1790526844582700 us; error 29 us; skew 500 ppm
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:04.585561 8916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:34:04.586561 8916 webserver.cc:460] Webserver started at http://127.0.98.3:39165/ using document root <none> and password file <none>
I20260927 16:34:04.587185 8916 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:04.587278 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:04.587507 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:04.589624 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "075c800c51074900896bd9d098a306c8"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "67e91b5d7e7c480ca728a3d463628c13"
server_key_iv: "3a75f4920edee5feb6eb8096b5fbd42e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.590234 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "075c800c51074900896bd9d098a306c8"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "67e91b5d7e7c480ca728a3d463628c13"
server_key_iv: "3a75f4920edee5feb6eb8096b5fbd42e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.593850 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:34:04.596297 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.597574 8916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:04.597728 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "075c800c51074900896bd9d098a306c8"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "67e91b5d7e7c480ca728a3d463628c13"
server_key_iv: "3a75f4920edee5feb6eb8096b5fbd42e"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.597833 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:04.619483 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:04.622357 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:04.622570 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:04.623193 8916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:04.624053 8916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:04.624143 8916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.624218 8916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:04.624262 8916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.633194 8916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38429
I20260927 16:34:04.633250 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38429 every 8 connection(s)
I20260927 16:34:04.634210 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:34:04.635547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 8916
I20260927 16:34:04.635677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:34:04.636030 392 external_mini_cluster.cc:1428] Setting key 4dc33177545662268d0289fe4948a639
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:04.647296 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.636288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49127 (local address 127.0.98.62:38445)
0927 16:34:04.636452 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:04.636456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:04.637486 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 16:34:04.639029 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:04.639031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:04.639075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:04.639153 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:04.640225 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.640722 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.641791 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.642002 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.642072 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.644376 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:04.644379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:04.644381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:04.644403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:04.645579 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.646007 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.646009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.646011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.646078 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:04.646423 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:04.646425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:04.646426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:04.646586 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:04.646659 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:04.647083 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 16:34:04.647184 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:34:04.648054 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:04.636738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:49127)
0927 16:34:04.637309 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:04.637364 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:04.638848 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:04.639285 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:04.639292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:04.639637 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:04.640110 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.640123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.640820 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.640822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.641693 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:04.641701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:04.642118 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:04.642121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:04.642273 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:04.642885 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:04.642900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:04.644238 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:04.645676 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.645680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.645690 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.645925 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.646159 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:04.646161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:04.646162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:04.646343 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:04.646670 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:04.646675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:04.646882 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 16:34:04.647025 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 16:34:04.647255 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":158,"threads_started":1}
I20260927 16:34:04.649263 9045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38445
I20260927 16:34:04.649495 9045 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:04.650018 9045 heartbeater.cc:511] Master 127.0.98.62:38445 requested a full tablet report, sending...
I20260927 16:34:04.651005 8586 ts_manager.cc:194] Registered new tserver with Master: 075c800c51074900896bd9d098a306c8 (127.0.98.3:38429)
I20260927 16:34:04.651506 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49127
I20260927 16:34:04.661847 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:34:04.732366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8648
I20260927 16:34:04.737910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8782
I20260927 16:34:04.743377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8916
I20260927 16:34:04.748826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 8554
I20260927 16:34:04.755214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38445
--webserver_interface=127.0.98.62
--webserver_port=46701
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:04.863070 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:04.863457 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:04.863539 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:04.867684 9054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:04.867781 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:04.867817 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:04.867856 9054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:04.867887 9054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:04.873359 9054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:04.874526 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:04.875586 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:04.884131 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:04.884171 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.884151 9054 server_base.cc:1034] running on GCE node
W20260927 16:34:04.884145 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:04.884859 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:04.885936 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:04.887089 9054 hybrid_clock.cc:648] HybridClock initialized: now 1790526844887050 us; error 50 us; skew 500 ppm
Sep 27 16:34:04 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:04.890421 9054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:34:04.891295 9054 webserver.cc:460] Webserver started at http://127.0.98.62:46701/ using document root <none> and password file <none>
I20260927 16:34:04.891812 9054 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:04.891880 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:04.895521 9054 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:04.897610 9069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:04.898851 9054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:04.898995 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8437510206fb474e8427c392f16bc021"
format_stamp: "Formatted at 2026-09-27 16:34:03 on dist-test-slave-t27c"
server_key: "07c6e0149f19a97061161f5dfe248a03"
server_key_iv: "71166958a2a0e053de8747ef383a29ea"
server_key_version: "encryptionkey@0"
I20260927 16:34:04.899509 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:04.927099 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:04.930125 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:04.930346 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:04.937089 9054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38445
I20260927 16:34:04.937129 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38445 every 8 connection(s)
I20260927 16:34:04.938211 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:34:04.940825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9054
I20260927 16:34:04.941244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33999
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38445
--webserver_port=37171
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:34:04.942669 9122 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:34:04.944674 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Bootstrap starting.
I20260927 16:34:04.968427 9122 log.cc:824] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:04.976146 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:34:04.976500 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Bootstrap complete.
I20260927 16:34:04.980561 9122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:04.980981 9122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8437510206fb474e8427c392f16bc021, State: Initialized, Role: FOLLOWER
I20260927 16:34:04.981488 9122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:04.981652 9122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:34:04.981734 9122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:34:04.981858 9122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:34:04.983945 9122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:04.984304 9122 leader_election.cc:302] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8437510206fb474e8427c392f16bc021; no voters:
I20260927 16:34:04.984655 9122 leader_election.cc:288] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:34:04.984800 9126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:34:04.985044 9126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [term 2 LEADER]: Becoming Leader. State: Replica: 8437510206fb474e8427c392f16bc021, State: Running, Role: LEADER
I20260927 16:34:04.985424 9126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } }
I20260927 16:34:04.985940 9122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:34:04.986965 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8437510206fb474e8427c392f16bc021. Latest consensus state: current_term: 2 leader_uuid: "8437510206fb474e8427c392f16bc021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } } }
I20260927 16:34:04.987105 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:04.987428 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8437510206fb474e8427c392f16bc021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8437510206fb474e8427c392f16bc021" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38445 } } }
I20260927 16:34:04.987469 9135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:34:04.987529 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:04.990785 9135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:34:04.991608 9135 catalog_manager.cc:1259] Loaded cluster ID: 30ab75fe9c1448f48884ce9d8d7620da
I20260927 16:34:04.991676 9135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:34:04.993989 9135 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:34:04.994630 9135 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8437510206fb474e8427c392f16bc021: Loaded TSK: 0
I20260927 16:34:04.995836 9135 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:34:05.069257 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:05.069521 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:05.069567 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:05.073745 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:05.073809 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:05.073869 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:34:05.079290 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37171
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:05.080482 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:05.081467 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:05.092120 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:05.092116 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:05.092368 9124 server_base.cc:1034] running on GCE node
W20260927 16:34:05.092460 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:05.092882 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:05.093564 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:05.094745 9124 hybrid_clock.cc:648] HybridClock initialized: now 1790526845094722 us; error 45 us; skew 500 ppm
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:05.097555 9124 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:34:05.098594 9124 webserver.cc:460] Webserver started at http://127.0.98.1:37171/ using document root <none> and password file <none>
I20260927 16:34:05.099153 9124 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:05.099256 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:05.103534 9124 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 16:34:05.106029 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.107504 9124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:34:05.107650 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "06eefd57b7cf4c4891ccf154f9fe8a37"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "e6f1c32d2740f1036f1baac3c5332681"
server_key_iv: "987e8b80263e1cb91fbf664bf4988508"
server_key_version: "encryptionkey@0"
I20260927 16:34:05.108280 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:05.120082 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:05.123323 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:05.123533 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:05.124158 9124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:05.124956 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:05.125027 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.125095 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:05.125150 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.135170 9124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260927 16:34:05.135193 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260927 16:34:05.136127 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:34:05.138422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9124
I20260927 16:34:05.138939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38445
--webserver_port=41967
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.151733 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.138349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:45957)
0927 16:34:05.139134 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.139189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.140778 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.142099 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.142106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.142543 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.143023 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.143038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.145160 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.145163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.146120 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.146160 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.147402 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.147406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.147548 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.148223 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.148239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.150938 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":120,"threads_started":1}
W20260927 16:34:05.152251 9272 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38445: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260927 16:34:05.152638 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.138225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45957 (local address 127.0.98.62:38445)
0927 16:34:05.138754 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.138774 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.139293 (+ 519us) server_negotiation.cc:407] Connection header received
0927 16:34:05.141230 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.141246 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.141640 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.141943 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.143222 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.144994 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.146345 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.147199 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.147392 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.151217 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45957: BlockingRecv error: recv got EOF from 127.0.98.1:45957 (error 108)
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":122,"threads_started":1}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.158401 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.152740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:56821)
0927 16:34:05.153107 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.153124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.153231 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.153512 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.153516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.153696 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.153792 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.153797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.154815 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.154817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.155354 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.155362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.155906 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.155909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.155991 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.156483 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.156494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.158313 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287}
W20260927 16:34:05.158564 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.152795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56821 (local address 127.0.98.62:38445)
0927 16:34:05.153227 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.153232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.153247 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:05.153330 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.153333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.153379 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.153460 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.153892 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.154717 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.155458 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.155797 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.155892 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.158453 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56821: BlockingRecv error: recv got EOF from 127.0.98.1:56821 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":186,"threads_started":1}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.164168 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.158909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:44245)
0927 16:34:05.159071 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.159084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.159154 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.159351 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.159353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.159490 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.159553 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.159557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.160515 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.160518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.161050 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.161058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.161606 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.161609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.161667 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.162142 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.162153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.164048 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 16:34:05.164332 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.158997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44245 (local address 127.0.98.62:38445)
0927 16:34:05.159108 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.159113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.159128 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:05.159215 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.159218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.159255 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.159335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.159648 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.160407 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.161156 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.161493 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.161595 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.164238 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44245: BlockingRecv error: recv got EOF from 127.0.98.1:44245 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 16:34:05.164592 9272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:34:05.257645 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:05.257966 9276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:05.258044 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:05.262192 9276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:05.262302 9276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:05.262399 9276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:34:05.268463 9276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41967
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:05.269640 9276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:05.270771 9276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:05.280175 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:05.280200 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:05.280233 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:05.280570 9276 server_base.cc:1034] running on GCE node
I20260927 16:34:05.280959 9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:05.281519 9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:05.282706 9276 hybrid_clock.cc:648] HybridClock initialized: now 1790526845282694 us; error 24 us; skew 500 ppm
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:05.285305 9276 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:34:05.286309 9276 webserver.cc:460] Webserver started at http://127.0.98.2:41967/ using document root <none> and password file <none>
I20260927 16:34:05.286988 9276 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:05.287087 9276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:05.290827 9276 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 16:34:05.292757 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.293888 9276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:05.294013 9276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "11822b15868948a288499d9589d21d87"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "4be031bd3eea6a0047a7d1e4185dbdb6"
server_key_iv: "8e734a9c6892609025b8fbff32e38c9e"
server_key_version: "encryptionkey@0"
I20260927 16:34:05.294590 9276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:05.344775 9276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:05.348335 9276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:05.348516 9276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:05.349059 9276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:05.349866 9276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:05.349915 9276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.349954 9276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:05.349978 9276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.358546 9276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38733
I20260927 16:34:05.358644 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38733 every 8 connection(s)
I20260927 16:34:05.359597 9276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:34:05.367062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9276
I20260927 16:34:05.367559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38429
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38445
--webserver_port=39165
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:34:05.370466 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.361402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51659 (local address 127.0.98.62:38445)
0927 16:34:05.361549 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.361554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.362454 (+ 900us) server_negotiation.cc:407] Connection header received
0927 16:34:05.363782 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.363784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.363832 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.363909 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.365046 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.365560 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.366665 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.366871 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.366955 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.370347 (+ 3392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51659: BlockingRecv error: recv got EOF from 127.0.98.2:51659 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:34:05.371116 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.361854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:51659)
0927 16:34:05.362310 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.362353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.363582 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.364081 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.364087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.364421 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.364880 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.364892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.365666 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.365669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.366560 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.366569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.367034 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.367039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.367237 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.367926 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.367941 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.370229 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
W20260927 16:34:05.371537 9406 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38445: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.377564 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.371993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:33827)
0927 16:34:05.372182 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.372193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.372262 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.372503 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.372505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.372623 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.372708 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.372712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.373796 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.373799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.374402 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.374410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.375032 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.375034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.375100 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.375588 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.375598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.377452 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 16:34:05.378043 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.372061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33827 (local address 127.0.98.62:38445)
0927 16:34:05.372153 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.372156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.372263 (+ 107us) server_negotiation.cc:407] Connection header received
0927 16:34:05.372328 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.372332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.372390 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.372487 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.372816 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.373684 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.374513 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.374922 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.375043 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.377650 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33827: BlockingRecv error: recv got EOF from 127.0.98.2:33827 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.383414 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.378243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:51117)
0927 16:34:05.378379 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.378397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.378477 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.378707 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.378710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.378861 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.378944 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.378950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.379617 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.379620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.380465 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.380473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.380862 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.380866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.380945 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.381386 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.381400 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.383282 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 16:34:05.383607 9406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:34:05.383786 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.378419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51117 (local address 127.0.98.62:38445)
0927 16:34:05.378531 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.378534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.378545 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:34:05.378573 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.378575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.378605 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.378669 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.379043 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.379522 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.380571 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.380763 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.380837 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.383418 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51117: BlockingRecv error: recv got EOF from 127.0.98.2:51117 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:34:05.484953 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:05.485201 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:05.485284 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:05.489691 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:05.489748 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:05.489856 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:34:05.495333 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39165
--tserver_master_addrs=127.0.98.62:38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:05.496392 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:05.497486 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:05.507520 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:05.507526 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:05.507552 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:05.507766 9409 server_base.cc:1034] running on GCE node
I20260927 16:34:05.508327 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:05.509016 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:05.510233 9409 hybrid_clock.cc:648] HybridClock initialized: now 1790526845510190 us; error 50 us; skew 500 ppm
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:05.513054 9409 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:34:05.513931 9409 webserver.cc:460] Webserver started at http://127.0.98.3:39165/ using document root <none> and password file <none>
I20260927 16:34:05.514525 9409 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:05.514599 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:05.518453 9409 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 16:34:05.520449 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.521668 9409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:05.521828 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "075c800c51074900896bd9d098a306c8"
format_stamp: "Formatted at 2026-09-27 16:34:04 on dist-test-slave-t27c"
server_key: "67e91b5d7e7c480ca728a3d463628c13"
server_key_iv: "3a75f4920edee5feb6eb8096b5fbd42e"
server_key_version: "encryptionkey@0"
I20260927 16:34:05.522406 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:05.545516 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:05.549026 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:05.549242 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:05.549881 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:05.550781 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:05.550853 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.550920 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:05.550974 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:05.559746 9409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38429
I20260927 16:34:05.559773 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38429 every 8 connection(s)
I20260927 16:34:05.560755 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:34:05.564319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9409
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:34:05.572453 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.562532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39905 (local address 127.0.98.62:38445)
0927 16:34:05.562679 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.562682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.563529 (+ 847us) server_negotiation.cc:407] Connection header received
0927 16:34:05.565007 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.565011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.565068 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.565166 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.566617 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.567150 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.568417 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.568761 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.568857 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.572329 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39905: BlockingRecv error: recv got EOF from 127.0.98.3:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:34:05.573225 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.562867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:39905)
0927 16:34:05.563353 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.563396 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.564767 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.565380 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.565388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.565882 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.566478 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.566492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.567275 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.567282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.568291 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.568300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.568975 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.568980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.569188 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.569703 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.569719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.572181 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
W20260927 16:34:05.573861 9538 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38445: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 16:34:05.574314 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.565446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.0.98.62:38445)
0927 16:34:05.565599 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.565612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.565627 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:05.565746 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.565749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.565799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.565898 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.566685 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.567226 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.568061 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.568391 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.568481 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.569945 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:05.569951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:05.569969 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:05.569998 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:05.571846 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:05.572342 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:05.572349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:05.572355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:05.572520 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:05.572881 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:05.572884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:05.572886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:05.573337 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:05.573520 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:05.573794 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 16:34:05.574117 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.579061 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.574346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:58021)
0927 16:34:05.574454 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.574470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.574552 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.574763 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.574766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.574921 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.575038 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.575044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.575758 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.575764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.576323 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.576332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.576853 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.576856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.576922 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.577373 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.577383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.578975 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:34:05.579208 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.574431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58021 (local address 127.0.98.62:38445)
0927 16:34:05.574532 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.574536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.574549 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:34:05.574629 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.574631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.574661 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.574722 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.575165 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.575632 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.576430 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.576732 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.576844 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.579095 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58021: BlockingRecv error: recv got EOF from 127.0.98.3:58021 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":45}
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:05 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:05.584177 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:05.579687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:49203)
0927 16:34:05.579795 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:05.579810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:05.579883 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:05.580106 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:05.580108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:05.580257 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:05.580318 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.580323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.580992 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.580999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.581523 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:05.581530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.581964 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.581968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:05.582044 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.582470 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:05.582481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:05.584084 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 16:34:05.584408 9538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:34:05.584430 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:05.579763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49203 (local address 127.0.98.62:38445)
0927 16:34:05.579894 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:05.579898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:05.579923 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:34:05.579957 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:05.579960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:05.579998 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:05.580092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:05.580423 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.580890 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.581638 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:05.581857 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:05.581964 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:05.584202 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49203: BlockingRecv error: recv got EOF from 127.0.98.3:49203 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:06.171852 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:06.165188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:48819)
0927 16:34:06.165467 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:06.165487 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:06.165612 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:06.165906 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:06.165912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:06.166147 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:06.166263 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.166268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.167013 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.167016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.167747 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.167760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.168344 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.168347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.168429 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.169004 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:06.169019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:06.171708 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
W20260927 16:34:06.172143 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:06.165290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48819 (local address 127.0.98.62:38445)
0927 16:34:06.165573 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:06.165577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:06.165589 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:34:06.165718 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:06.165721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:06.165772 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:06.165888 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:06.166368 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.166893 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.167876 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.168209 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.168340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.171883 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48819: BlockingRecv error: recv got EOF from 127.0.98.1:48819 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":58,"threads_started":1}
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:06.390892 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:06.384322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:41121)
0927 16:34:06.384596 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:06.384610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:06.384711 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:06.384974 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:06.385167 (+ 193us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:06.385383 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:06.385511 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.385518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.386476 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.386479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.387248 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.387263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.387897 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.387900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.387976 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.388576 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:06.388588 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:06.390741 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
W20260927 16:34:06.391048 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:06.384448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41121 (local address 127.0.98.62:38445)
0927 16:34:06.384641 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:06.384645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:06.384664 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:34:06.384811 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:06.384813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:06.384859 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:06.384932 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:06.385674 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.386335 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.387408 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.387714 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.387818 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.390916 (+ 3098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41121: BlockingRecv error: recv got EOF from 127.0.98.2:41121 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:06 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:06.591030 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:06.585189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:40425)
0927 16:34:06.585539 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:06.585556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:06.585680 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:06.585910 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:06.585912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:06.586091 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:06.586218 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.586223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.587058 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.587061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.587752 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:06.587764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.588258 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.588261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:06.588330 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.588833 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:06.588845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:06.590839 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":176,"threads_started":1}
W20260927 16:34:06.591059 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:06.585246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40425 (local address 127.0.98.62:38445)
0927 16:34:06.585400 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:06.585403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:06.585648 (+ 245us) server_negotiation.cc:407] Connection header received
0927 16:34:06.585762 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:06.585764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:06.585824 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:06.585892 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:06.586325 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.586973 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.587904 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:06.588127 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:06.588204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:06.590957 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40425: BlockingRecv error: recv got EOF from 127.0.98.3:40425 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:07.179101 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:07.172715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:46919)
0927 16:34:07.173046 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:07.173065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:07.173173 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:07.173781 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:07.173784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:07.173998 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:07.174090 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.174095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.175264 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.175271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.175907 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.175916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.176404 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.176408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.176485 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.177122 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:07.177136 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:07.178963 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
W20260927 16:34:07.179234 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:07.173295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46919 (local address 127.0.98.62:38445)
0927 16:34:07.173539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:07.173543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:07.173558 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:07.173599 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:07.173601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:07.173647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:07.173744 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:07.174226 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.175114 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.176038 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.176284 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.176399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.179099 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46919: BlockingRecv error: recv got EOF from 127.0.98.1:46919 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:07.397712 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:07.391919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:51735)
0927 16:34:07.392210 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:07.392224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:07.392333 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:07.392594 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:07.392596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:07.392786 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:07.392883 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.392889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.393608 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.393611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.394263 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.394273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.394789 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.394794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.394879 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.395554 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:07.395569 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:07.397522 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260927 16:34:07.397753 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:07.391986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51735 (local address 127.0.98.62:38445)
0927 16:34:07.392139 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:07.392143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:07.392299 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:34:07.392421 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:07.392424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:07.392470 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:07.392550 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:07.392991 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.393505 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.394403 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.394648 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.394761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.397649 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51735: BlockingRecv error: recv got EOF from 127.0.98.2:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:07 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:07.602985 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:07.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:58031)
0927 16:34:07.592211 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:07.592225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:07.592325 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:07.592576 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:07.592578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:07.592752 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:07.592862 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.592867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.593576 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.593578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.594337 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:07.594349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.594848 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.594860 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:07.594938 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.600958 (+ 6020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:07.600977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:07.602841 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
W20260927 16:34:07.603147 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:07.592091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58031 (local address 127.0.98.62:38445)
0927 16:34:07.592222 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:07.592226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:07.592293 (+ 67us) server_negotiation.cc:407] Connection header received
0927 16:34:07.592416 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:07.592419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:07.592464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:07.592538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:07.592957 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.593484 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.594464 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:07.594722 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:07.594837 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:07.602987 (+ 8150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58031: BlockingRecv error: recv got EOF from 127.0.98.3:58031 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:08.187242 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:08.180135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:56093)
0927 16:34:08.180411 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:08.180426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:08.180538 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:08.180867 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:08.180872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:08.181102 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:08.181231 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.181239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.182431 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.182435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.183337 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.183348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.184034 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.184038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.184130 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.184758 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:08.184774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:08.187053 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
W20260927 16:34:08.187403 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:08.180191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56093 (local address 127.0.98.62:38445)
0927 16:34:08.180486 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:08.180490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:08.180504 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:08.180665 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:08.180667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:08.180709 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:08.180798 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:08.181363 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.182323 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.183461 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.183894 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.184018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.187269 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56093: BlockingRecv error: recv got EOF from 127.0.98.1:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":135,"threads_started":1}
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:08.404181 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:08.398477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:46601)
0927 16:34:08.398795 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:08.398808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:08.398934 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:08.399203 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:08.399205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:08.399385 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:08.399493 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.399498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.400252 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.400254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.400841 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.400848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.401323 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.401326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.401387 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.401982 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:08.401993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:08.404017 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
W20260927 16:34:08.404382 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:08.398549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46601 (local address 127.0.98.62:38445)
0927 16:34:08.398692 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:08.398696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:08.398862 (+ 166us) server_negotiation.cc:407] Connection header received
0927 16:34:08.399029 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:08.399031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:08.399073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:08.399169 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:08.399594 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.400134 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.400982 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.401215 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.401306 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.404178 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46601: BlockingRecv error: recv got EOF from 127.0.98.2:46601 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:08 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:08.609931 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:08.603839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:46391)
0927 16:34:08.604172 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:08.604186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:08.604296 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:08.604600 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:08.604603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:08.604812 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:08.604909 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.604915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.605871 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.605878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.606582 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:08.606594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.607077 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.607080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:08.607180 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.607718 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:08.607730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:08.609714 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
W20260927 16:34:08.610064 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:08.603914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46391 (local address 127.0.98.62:38445)
0927 16:34:08.604071 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:08.604076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:08.604356 (+ 280us) server_negotiation.cc:407] Connection header received
0927 16:34:08.604419 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:08.604424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:08.604480 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:08.604563 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:08.605061 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.605698 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.606708 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:08.606939 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:08.607036 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:08.609926 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46391: BlockingRecv error: recv got EOF from 127.0.98.3:46391 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:09.193856 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:09.188207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:51423)
0927 16:34:09.188516 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:09.188530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:09.188622 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:09.188856 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:09.188858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:09.189041 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:09.189137 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.189142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.189879 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.189882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.190521 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.190531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.190953 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.190956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.191018 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.191494 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:09.191504 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:09.193729 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
W20260927 16:34:09.194059 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:09.188280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51423 (local address 127.0.98.62:38445)
0927 16:34:09.188556 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:09.188560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:09.188574 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:09.188706 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:09.188708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:09.188766 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:09.188846 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:09.189231 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.189778 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.190621 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.190853 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.190932 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.193885 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51423: BlockingRecv error: recv got EOF from 127.0.98.1:51423 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:09.411515 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:09.405018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:34271)
0927 16:34:09.405312 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:09.405326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:09.405476 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:09.405765 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:09.405768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:09.405982 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:09.406086 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.406092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.406945 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.406951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.407561 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.407573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.408202 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.408206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.408299 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.408880 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:09.408892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:09.411337 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20260927 16:34:09.411594 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:09.405065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34271 (local address 127.0.98.62:38445)
0927 16:34:09.405260 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:09.405265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:09.405410 (+ 145us) server_negotiation.cc:407] Connection header received
0927 16:34:09.405588 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:09.405592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:09.405645 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:09.405739 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:09.406220 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.406810 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.407704 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.408058 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.408178 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.411480 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34271: BlockingRecv error: recv got EOF from 127.0.98.2:34271 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:09 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:09.616328 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:09.610779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:36571)
0927 16:34:09.611054 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:09.611067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:09.611184 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:09.611515 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:09.611517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:09.611772 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:09.611871 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.611877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.612782 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.612787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.613483 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:09.613492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.613904 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.613906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:09.613963 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.614436 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:09.614447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:09.616214 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":111,"threads_started":1}
W20260927 16:34:09.616492 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:09.610823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36571 (local address 127.0.98.62:38445)
0927 16:34:09.610961 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:09.610965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:09.611137 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:34:09.611315 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:09.611317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:09.611386 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:09.611475 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:09.611951 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.612648 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.613588 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:09.613811 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:09.613889 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:09.616337 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36571: BlockingRecv error: recv got EOF from 127.0.98.3:36571 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:10.200531 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:10.194776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:58565)
0927 16:34:10.195072 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:10.195086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:10.195187 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:10.195437 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:10.195439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:10.195640 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:10.195733 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.195739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.196456 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.196458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.197031 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.197039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.197597 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.197600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.197682 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.198338 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:10.198352 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:10.200407 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
W20260927 16:34:10.200680 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:10.194901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58565 (local address 127.0.98.62:38445)
0927 16:34:10.195148 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:10.195153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:10.195169 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:34:10.195277 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:10.195280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:10.195323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:10.195404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:10.195836 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.196366 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.197159 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.197476 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.197583 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.200542 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58565: BlockingRecv error: recv got EOF from 127.0.98.1:58565 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:10.417850 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:10.412448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:51137)
0927 16:34:10.412732 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:10.412752 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:10.412845 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:10.413052 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:10.413053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:10.413226 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:10.413318 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.413323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.414051 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.414053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.414659 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.414668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.415233 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.415236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.415303 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.415815 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:10.415825 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:10.417722 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
W20260927 16:34:10.418023 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:10.412605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51137 (local address 127.0.98.62:38445)
0927 16:34:10.412742 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:10.412746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:10.412822 (+ 76us) server_negotiation.cc:407] Connection header received
0927 16:34:10.412924 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:10.412926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:10.412965 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:10.413040 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:10.413417 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.413948 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.414829 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.415113 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.415202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.417867 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51137: BlockingRecv error: recv got EOF from 127.0.98.2:51137 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:10 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:10.622313 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:10.617179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:60661)
0927 16:34:10.617438 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:10.617451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:10.617539 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:10.617773 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:10.617776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:10.617950 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:10.618042 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.618047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.618778 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.618780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.619346 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:10.619354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.619763 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.619765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:10.619823 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.620387 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:10.620398 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:10.622136 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
W20260927 16:34:10.622507 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:10.617225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60661 (local address 127.0.98.62:38445)
0927 16:34:10.617353 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:10.617357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:10.617516 (+ 159us) server_negotiation.cc:407] Connection header received
0927 16:34:10.617623 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:10.617625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:10.617668 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:10.617735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:10.618138 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.618687 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.619444 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:10.619670 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:10.619746 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:10.622344 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60661: BlockingRecv error: recv got EOF from 127.0.98.3:60661 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:11.206622 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:11.201310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:50613)
0927 16:34:11.201613 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:11.201626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:11.201714 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:11.201957 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:11.201959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:11.202148 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:11.202300 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.202305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.203030 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.203033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.203588 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.203595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.204023 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.204026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.204101 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.204670 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:11.204681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:11.206495 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
W20260927 16:34:11.206897 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:11.201400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50613 (local address 127.0.98.62:38445)
0927 16:34:11.201665 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:11.201668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:11.201682 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:34:11.201796 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:11.201798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:11.201840 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:11.201921 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:11.202407 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.202938 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.203686 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.203921 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.203999 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.206759 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50613: BlockingRecv error: recv got EOF from 127.0.98.1:50613 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:11.424305 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:11.418700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:53421)
0927 16:34:11.418983 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:11.418997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:11.419130 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:11.419408 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:11.419411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:11.419658 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:11.419776 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.419783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.420548 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.420552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.421120 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.421127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.421614 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.421618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.421703 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.422231 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:11.422242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:11.424133 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20260927 16:34:11.424374 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:11.419068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53421 (local address 127.0.98.62:38445)
0927 16:34:11.419186 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:11.419193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:11.419210 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:34:11.419246 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:11.419248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:11.419287 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:11.419383 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:11.419901 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.420432 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.421239 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.421502 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.421614 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.424264 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53421: BlockingRecv error: recv got EOF from 127.0.98.2:53421 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:11 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:11.628530 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:11.623082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:57677)
0927 16:34:11.623331 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:11.623344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:11.623445 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:11.623671 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:11.623673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:11.623846 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:11.623949 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.623955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.624673 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.624676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.625273 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:11.625281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.625758 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.625760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:11.625828 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.626393 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:11.626405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:11.628403 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
W20260927 16:34:11.628687 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:11.623169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57677 (local address 127.0.98.62:38445)
0927 16:34:11.623336 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:11.623340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:11.623413 (+ 73us) server_negotiation.cc:407] Connection header received
0927 16:34:11.623524 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:11.623526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:11.623571 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:11.623636 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:11.624043 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.624573 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.625405 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:11.625657 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:11.625741 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:11.628562 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57677: BlockingRecv error: recv got EOF from 127.0.98.3:57677 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:12.212659 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:12.207419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:33285)
0927 16:34:12.207730 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:12.207743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:12.207839 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:12.208075 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:12.208078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:12.208254 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:12.208347 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.208353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.209091 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.209093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.209653 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.209661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.210057 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.210059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.210117 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.210714 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:12.210725 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:12.212540 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":69,"threads_started":1}
W20260927 16:34:12.212814 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:12.207477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33285 (local address 127.0.98.62:38445)
0927 16:34:12.207744 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:12.207747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:12.207793 (+ 46us) server_negotiation.cc:407] Connection header received
0927 16:34:12.207922 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:12.207924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:12.207967 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:12.208041 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:12.208446 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.208990 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.209753 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.209963 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.210037 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.212677 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33285: BlockingRecv error: recv got EOF from 127.0.98.1:33285 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:12.431071 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:12.425150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:43169)
0927 16:34:12.425521 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:12.425534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:12.425663 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:12.425951 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:12.425954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:12.426215 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:12.426354 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.426362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.427205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.427212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.427803 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.427812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.428302 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.428305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.428392 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.428961 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:12.428974 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:12.430943 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
W20260927 16:34:12.431370 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:12.425227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43169 (local address 127.0.98.62:38445)
0927 16:34:12.425390 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:12.425395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:12.425611 (+ 216us) server_negotiation.cc:407] Connection header received
0927 16:34:12.425782 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:12.425785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:12.425842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:12.425924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:12.426496 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.427084 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.427930 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.428181 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.428292 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.431132 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43169: BlockingRecv error: recv got EOF from 127.0.98.2:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:12 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:12.639816 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:12.629597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:40921)
0927 16:34:12.629934 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:12.629949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:12.630057 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:12.630346 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:12.630348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:12.630541 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:12.630637 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.630642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.631398 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.631400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.631962 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:12.631970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.632390 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.632392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:12.632456 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.637799 (+ 5343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:12.637812 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:12.639699 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
W20260927 16:34:12.639953 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:12.629685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40921 (local address 127.0.98.62:38445)
0927 16:34:12.629835 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:12.629840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:12.630025 (+ 185us) server_negotiation.cc:407] Connection header received
0927 16:34:12.630142 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:12.630144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:12.630234 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:12.630305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:12.630749 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.631302 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.632071 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:12.632290 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:12.632369 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:12.639834 (+ 7465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40921: BlockingRecv error: recv got EOF from 127.0.98.3:40921 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:13.219550 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:13.213547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:39037)
0927 16:34:13.213864 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:13.213877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:13.213972 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:13.214370 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:13.214373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:13.214571 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:13.214694 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.214701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.215491 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.215494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.216159 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.216169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.216680 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.216683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.216760 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.217358 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:13.217371 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:13.219353 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":126,"threads_started":1}
W20260927 16:34:13.219610 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:13.213678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39037 (local address 127.0.98.62:38445)
0927 16:34:13.213915 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:13.213919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:13.213934 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:34:13.214053 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:13.214055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:13.214098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:13.214229 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:13.214819 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.215371 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.216293 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.216546 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.216631 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.219487 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39037: BlockingRecv error: recv got EOF from 127.0.98.1:39037 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:13.438123 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:13.431982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:43927)
0927 16:34:13.432267 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:13.432291 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:13.432400 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:13.432651 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:13.432653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:13.432864 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:13.432965 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.432970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.433993 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.433997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.434749 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.434761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.435380 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.435384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.435477 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.435980 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:13.435992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:13.438006 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
W20260927 16:34:13.438369 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:13.432078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43927 (local address 127.0.98.62:38445)
0927 16:34:13.432252 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:13.432256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:13.432375 (+ 119us) server_negotiation.cc:407] Connection header received
0927 16:34:13.432484 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:13.432486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:13.432530 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:13.432612 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:13.433074 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.433831 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.434879 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.435211 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.435334 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.438145 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43927: BlockingRecv error: recv got EOF from 127.0.98.2:43927 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:13 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:13.645965 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:13.640672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:47291)
0927 16:34:13.640951 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:13.640965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:13.641060 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:13.641306 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:13.641308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:13.641491 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:13.641586 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.641592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.642391 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.642394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.642963 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:13.642970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.643420 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.643423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:13.643480 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.643951 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:13.643962 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:13.645786 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
W20260927 16:34:13.646001 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:13.640825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47291 (local address 127.0.98.62:38445)
0927 16:34:13.640963 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:13.640967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:13.641029 (+ 62us) server_negotiation.cc:407] Connection header received
0927 16:34:13.641151 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:13.641154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:13.641199 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:13.641269 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:13.641690 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.642271 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.643083 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:13.643321 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:13.643403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:13.645901 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47291: BlockingRecv error: recv got EOF from 127.0.98.3:47291 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:14.226293 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:14.220546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:59423)
0927 16:34:14.220884 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:14.220901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:14.221020 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:14.221350 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:14.221353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:14.221558 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:14.221648 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.221654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.222493 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.222500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.223066 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.223074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.223537 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.223541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.223629 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.224099 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:14.224110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:14.226082 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
W20260927 16:34:14.226428 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:14.220640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59423 (local address 127.0.98.62:38445)
0927 16:34:14.220894 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:14.220898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:14.220979 (+ 81us) server_negotiation.cc:407] Connection header received
0927 16:34:14.221156 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:14.221161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:14.221217 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:14.221326 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:14.221762 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.222386 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.223187 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.223424 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.223535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.226286 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59423: BlockingRecv error: recv got EOF from 127.0.98.1:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:14.445067 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:14.439053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:34869)
0927 16:34:14.439350 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:14.439363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:14.439473 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:14.439731 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:14.439733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:14.439913 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:14.440008 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.440013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.440742 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.440744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.441318 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.441326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.441767 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.441770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.441837 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.442466 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:14.442489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:14.444872 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260927 16:34:14.445220 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:14.439176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34869 (local address 127.0.98.62:38445)
0927 16:34:14.439349 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:14.439353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:14.439441 (+ 88us) server_negotiation.cc:407] Connection header received
0927 16:34:14.439551 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:14.439553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:14.439597 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:14.439698 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:14.440104 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.440620 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.441420 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.441660 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.441740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.445060 (+ 3320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34869: BlockingRecv error: recv got EOF from 127.0.98.2:34869 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:14 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:14.651979 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:14.646899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:34305)
0927 16:34:14.647191 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:14.647204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:14.647320 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:14.647559 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:14.647561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:14.647736 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:14.647830 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.647835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.648543 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.648545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.649081 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:14.649088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.649486 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.649489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:14.649549 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.650029 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:14.650039 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:14.651865 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
W20260927 16:34:14.652130 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:14.646990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34305 (local address 127.0.98.62:38445)
0927 16:34:14.647151 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:14.647155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:14.647277 (+ 122us) server_negotiation.cc:407] Connection header received
0927 16:34:14.647402 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:14.647405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:14.647450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:14.647522 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:14.647925 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.648447 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.649177 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:14.649389 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:14.649464 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:14.651994 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34305: BlockingRecv error: recv got EOF from 127.0.98.3:34305 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:15.232901 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:15.227142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:52397)
0927 16:34:15.227457 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:15.227474 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:15.227579 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:15.227920 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:15.227923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:15.228124 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:15.228224 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.228229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.229003 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.229009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.229584 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.229592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.230063 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.230066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.230152 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.230795 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:15.230807 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:15.232769 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260927 16:34:15.233189 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:15.227196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52397 (local address 127.0.98.62:38445)
0927 16:34:15.227675 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:15.227704 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:15.227716 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:34:15.227749 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:15.227751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:15.227792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:15.227893 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:15.228343 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.228894 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.229721 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.229948 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.230059 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.232943 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52397: BlockingRecv error: recv got EOF from 127.0.98.1:52397 (error 108)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":92,"threads_started":1}
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:15.451702 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:15.445872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:36771)
0927 16:34:15.446247 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:15.446261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:15.446393 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:15.446673 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:15.446676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:15.446885 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:15.446971 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.446977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.447781 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.447786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.448367 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.448375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.448854 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.448858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.448942 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.449471 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:15.449483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:15.451578 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
W20260927 16:34:15.451866 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:15.445984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36771 (local address 127.0.98.62:38445)
0927 16:34:15.446192 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:15.446197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:15.446349 (+ 152us) server_negotiation.cc:407] Connection header received
0927 16:34:15.446506 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:15.446510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:15.446559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:15.446648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:15.447112 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.447622 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.448515 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.448742 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.448851 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.451717 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36771: BlockingRecv error: recv got EOF from 127.0.98.2:36771 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:15 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:15.658702 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:15.653027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:38939)
0927 16:34:15.653432 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:15.653448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:15.653576 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:15.653850 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:15.653853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:15.654057 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:15.654145 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.654150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.654919 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.654925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.655486 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:15.655494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.655953 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.655957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:15.656043 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.656502 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:15.656512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:15.658528 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
W20260927 16:34:15.658788 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:15.653184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38939 (local address 127.0.98.62:38445)
0927 16:34:15.653428 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:15.653432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:15.653530 (+ 98us) server_negotiation.cc:407] Connection header received
0927 16:34:15.653677 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:15.653681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:15.653730 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:15.653826 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:15.654287 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.654811 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.655603 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:15.655840 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:15.655949 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:15.658664 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38939: BlockingRecv error: recv got EOF from 127.0.98.3:38939 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:16.239845 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:16.233856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:50217)
0927 16:34:16.234214 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:16.234233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:16.234344 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:16.234669 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:16.234672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:16.234904 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:16.234997 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.235002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.235795 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.235800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.236358 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.236365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.236826 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.236830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.236914 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.237442 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:16.237452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:16.239647 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260927 16:34:16.239920 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:16.233959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50217 (local address 127.0.98.62:38445)
0927 16:34:16.234268 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:16.234272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:16.234308 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:34:16.234470 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:16.234473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:16.234530 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:16.234626 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:16.235141 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.235685 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.236476 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.236713 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.236823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.239788 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50217: BlockingRecv error: recv got EOF from 127.0.98.1:50217 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:16.458423 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:16.452519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:44935)
0927 16:34:16.452851 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:16.452869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:16.453013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:16.453295 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:16.453298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:16.453505 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:16.453593 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.453598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.454472 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.454478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.455041 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.455048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.455528 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.455532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.455625 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.456218 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:16.456232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:16.458253 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
W20260927 16:34:16.458467 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:16.452657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44935 (local address 127.0.98.62:38445)
0927 16:34:16.452817 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:16.452822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:16.452954 (+ 132us) server_negotiation.cc:407] Connection header received
0927 16:34:16.453122 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:16.453125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:16.453172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:16.453269 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:16.453733 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.454366 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.455170 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.455412 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.455523 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.458366 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44935: BlockingRecv error: recv got EOF from 127.0.98.2:44935 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:16 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:16.665051 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:16.659480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:55959)
0927 16:34:16.659747 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:16.659761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:16.659850 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:16.660129 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:16.660132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:16.660313 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:16.660411 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.660417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.661193 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.661195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.661749 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:16.661756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.662272 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.662275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:16.662388 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.662901 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:16.662912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:16.664895 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
W20260927 16:34:16.665237 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:16.659668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55959 (local address 127.0.98.62:38445)
0927 16:34:16.659797 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:16.659801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:16.659814 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:34:16.659949 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:16.659951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:16.659998 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:16.660077 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:16.660536 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.661097 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.661859 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:16.662100 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:16.662279 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:16.665102 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55959: BlockingRecv error: recv got EOF from 127.0.98.3:55959 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:17.246891 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:17.240905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:56251)
0927 16:34:17.241395 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:17.241412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:17.241540 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:17.241966 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:17.241969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:17.242229 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:17.242334 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.242340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.243245 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.243248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.243796 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.243804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.244199 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.244201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.244260 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.244734 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:17.244744 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:17.246713 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
W20260927 16:34:17.246958 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:17.240991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56251 (local address 127.0.98.62:38445)
0927 16:34:17.241286 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:17.241290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:17.241499 (+ 209us) server_negotiation.cc:407] Connection header received
0927 16:34:17.241696 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:17.241699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:17.241756 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:17.241874 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:17.242455 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.243145 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.243893 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.244106 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.244180 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.246837 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56251: BlockingRecv error: recv got EOF from 127.0.98.1:56251 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:17.464589 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:17.459338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:57855)
0927 16:34:17.459639 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:17.459652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:17.459750 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:17.459973 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:17.459975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:17.460167 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:17.460257 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.460262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.460964 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.460966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.461522 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.461529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.461918 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.461920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.461978 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.462493 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:17.462504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:17.464420 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":125,"threads_started":1}
W20260927 16:34:17.464653 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:17.459402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57855 (local address 127.0.98.62:38445)
0927 16:34:17.459557 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:17.459562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:17.459722 (+ 160us) server_negotiation.cc:407] Connection header received
0927 16:34:17.459828 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:17.459830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:17.459874 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:17.459941 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:17.460349 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.460877 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.461620 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.461827 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.461901 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.464546 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57855: BlockingRecv error: recv got EOF from 127.0.98.2:57855 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:17 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:17.676802 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:17.665982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:38149)
0927 16:34:17.666345 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:17.666364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:17.666502 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:17.666775 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:17.666778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:17.666984 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:17.667083 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.667088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.667839 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.667845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.668405 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:17.668413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.668866 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.668870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:17.668954 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.674760 (+ 5806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:17.674774 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:17.676681 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
W20260927 16:34:17.677006 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:17.666138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38149 (local address 127.0.98.62:38445)
0927 16:34:17.666298 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:17.666303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:17.666440 (+ 137us) server_negotiation.cc:407] Connection header received
0927 16:34:17.666609 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:17.666613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:17.666666 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:17.666752 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:17.667204 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.667727 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.668533 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:17.668754 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:17.668864 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:17.676860 (+ 7996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38149: BlockingRecv error: recv got EOF from 127.0.98.3:38149 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:18.252849 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:18.247633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:42117)
0927 16:34:18.247992 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:18.248010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:18.248118 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:18.248366 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:18.248369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:18.248547 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:18.248644 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.248649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.249377 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.249379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.249927 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.249934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.250392 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.250394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.250459 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.250923 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:18.250934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:18.252732 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":67,"threads_started":1}
W20260927 16:34:18.253015 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:18.247751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42117 (local address 127.0.98.62:38445)
0927 16:34:18.248027 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:18.248031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:18.248065 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:34:18.248188 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:18.248190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:18.248231 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:18.248305 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:18.248749 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.249284 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.250052 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.250296 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.250373 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.252864 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42117: BlockingRecv error: recv got EOF from 127.0.98.1:42117 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:18.470925 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:18.465305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:40173)
0927 16:34:18.465636 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:18.465653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:18.465796 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:18.466063 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:18.466066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:18.466315 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:18.466407 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.466412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.467208 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.467214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.467781 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.467790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.468231 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.468235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.468328 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.468903 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:18.468915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:18.470761 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
W20260927 16:34:18.471061 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:18.465476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40173 (local address 127.0.98.62:38445)
0927 16:34:18.465629 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:18.465634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:18.465733 (+ 99us) server_negotiation.cc:407] Connection header received
0927 16:34:18.465895 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:18.465899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:18.465952 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:18.466042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:18.466526 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.467091 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.467900 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.468120 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.468228 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.470949 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40173: BlockingRecv error: recv got EOF from 127.0.98.2:40173 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:18 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:18.683516 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:18.677605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:43873)
0927 16:34:18.677956 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:18.677973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:18.678126 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:18.678461 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:18.678464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:18.678668 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:18.678758 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.678764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.679524 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.679529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.680085 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:18.680093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.680543 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.680547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:18.680630 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.681270 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:18.681281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:18.683343 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":99,"threads_started":1}
W20260927 16:34:18.683554 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:18.677745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43873 (local address 127.0.98.62:38445)
0927 16:34:18.677943 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:18.677948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:18.678064 (+ 116us) server_negotiation.cc:407] Connection header received
0927 16:34:18.678271 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:18.678274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:18.678325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:18.678416 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:18.678882 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.679414 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.680206 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:18.680430 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:18.680540 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:18.683456 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43873: BlockingRecv error: recv got EOF from 127.0.98.3:43873 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:19.259347 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:19.253828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:35589)
0927 16:34:19.254127 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:19.254140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:19.254287 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:19.254601 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:19.254606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:19.254842 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:19.254942 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.254948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.255901 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.255904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.256467 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.256474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.256881 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.256884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.256943 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.257420 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:19.257432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:19.259221 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
W20260927 16:34:19.259488 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:19.253903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35589 (local address 127.0.98.62:38445)
0927 16:34:19.254207 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:19.254211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:19.254233 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:34:19.254390 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:19.254394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:19.254448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:19.254555 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:19.255069 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.255768 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.256568 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.256782 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.256859 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.259341 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35589: BlockingRecv error: recv got EOF from 127.0.98.1:35589 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:19.477125 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:19.471769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:45747)
0927 16:34:19.472063 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:19.472077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:19.472176 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:19.472424 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:19.472426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:19.472595 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:19.472684 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.472689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.473406 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.473408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.473974 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.473981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.474425 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.474427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.474496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.475088 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:19.475100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:19.476970 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
W20260927 16:34:19.477232 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:19.471825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45747 (local address 127.0.98.62:38445)
0927 16:34:19.471980 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:19.471985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:19.472151 (+ 166us) server_negotiation.cc:407] Connection header received
0927 16:34:19.472255 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:19.472258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:19.472302 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:19.472392 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:19.472778 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.473295 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.474081 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.474318 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.474411 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.477100 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45747: BlockingRecv error: recv got EOF from 127.0.98.2:45747 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:19 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:19.689888 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:19.684392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.3:47695)
0927 16:34:19.684661 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:19.684674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:19.684772 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:19.685011 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:19.685013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:19.685190 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:19.685282 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.685287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.686101 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.686104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.686771 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:19.686780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.687176 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.687179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:19.687239 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.687819 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:19.687841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:19.689681 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
W20260927 16:34:19.689889 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:19.684483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47695 (local address 127.0.98.62:38445)
0927 16:34:19.684605 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:19.684610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:19.684739 (+ 129us) server_negotiation.cc:407] Connection header received
0927 16:34:19.684855 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:19.684858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:19.684904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:19.684973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:19.685389 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.686006 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.686867 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:19.687076 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:19.687153 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:19.689792 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47695: BlockingRecv error: recv got EOF from 127.0.98.3:47695 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 16:34:20 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:20 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:20.265681 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:20.260293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.1:57467)
0927 16:34:20.260646 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:20.260669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:20.260790 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:20.261105 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:20.261110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:20.261365 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:20.261491 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:20.261499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.262261 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.262263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:20.262834 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:20.262842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.263246 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.263248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:20.263322 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:20.263800 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:20.263811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:20.265556 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
W20260927 16:34:20.265839 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:20.260364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57467 (local address 127.0.98.62:38445)
0927 16:34:20.260693 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:20.260699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:20.260742 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:34:20.260900 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:20.260903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:20.260964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:20.261064 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:20.261612 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.262163 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.262935 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.263151 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.263228 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:20.265696 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57467: BlockingRecv error: recv got EOF from 127.0.98.1:57467 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
Sep 27 16:34:20 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:34:20 dist-test-slave-t27c krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:34:20.484342 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:20.478343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38445 (local address 127.0.98.2:60171)
0927 16:34:20.478670 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:20.478689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:20.478855 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:20.479147 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:20.479150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:20.479383 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:20.479481 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:20.479487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.480282 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.480286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:20.481096 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:20.481104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.481712 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.481716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:20.481805 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:20.482414 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:20.482425 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:20.484202 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
W20260927 16:34:20.484580 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:20.478498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60171 (local address 127.0.98.62:38445)
0927 16:34:20.478666 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:20.478672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:20.478794 (+ 122us) server_negotiation.cc:407] Connection header received
0927 16:34:20.478970 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:20.478973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:20.479025 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:20.479115 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:20.479612 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.480156 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.481246 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.481588 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.481695 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:20.484346 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60171: BlockingRecv error: recv got EOF from 127.0.98.2:60171 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:34:20.572382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9124
I20260927 16:34:20.578263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9276
I20260927 16:34:20.583588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9409
I20260927 16:34:20.588970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9054
2026-09-27T16:34:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18075 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T16:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:20Z Disabled control of system clock
I20260927 16:34:20.619328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37069
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:34:20.730134 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:20.730418 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:20.730451 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:20.734686 9604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:20.734750 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:20.734766 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:20.734783 9604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:20.734824 9604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:20.739779 9604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:20.740880 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:20.742293 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:20.751029 9609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:20.751029 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:20.751089 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:20.751315 9604 server_base.cc:1034] running on GCE node
I20260927 16:34:20.751986 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:20.753199 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:20.754390 9604 hybrid_clock.cc:648] HybridClock initialized: now 1790526860754374 us; error 33 us; skew 500 ppm
I20260927 16:34:20.756291 9604 webserver.cc:460] Webserver started at http://127.0.98.62:37177/ using document root <none> and password file <none>
I20260927 16:34:20.756947 9604 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:20.757040 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:20.757278 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:20.759428 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3"
format_stamp: "Formatted at 2026-09-27 16:34:20 on dist-test-slave-t27c"
server_key: "160a2f54edcbedf40278763265dbbd9e"
server_key_iv: "de9f573b9f1311ffae8749b951ce4de8"
server_key_version: "encryptionkey@0"
I20260927 16:34:20.760000 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3"
format_stamp: "Formatted at 2026-09-27 16:34:20 on dist-test-slave-t27c"
server_key: "160a2f54edcbedf40278763265dbbd9e"
server_key_iv: "de9f573b9f1311ffae8749b951ce4de8"
server_key_version: "encryptionkey@0"
I20260927 16:34:20.763741 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:34:20.766144 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:20.767462 9604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:20.767613 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3"
format_stamp: "Formatted at 2026-09-27 16:34:20 on dist-test-slave-t27c"
server_key: "160a2f54edcbedf40278763265dbbd9e"
server_key_iv: "de9f573b9f1311ffae8749b951ce4de8"
server_key_version: "encryptionkey@0"
I20260927 16:34:20.767730 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:20.791602 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:20.792397 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:20.792604 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:20.799505 9604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37069
I20260927 16:34:20.799517 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37069 every 8 connection(s)
I20260927 16:34:20.800473 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:34:20.803555 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:20.805029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9604
I20260927 16:34:20.805141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:34:20.805473 392 external_mini_cluster.cc:1428] Setting key 3c20057ec7e1c7de28525c184ff197b4
I20260927 16:34:20.809859 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3: Bootstrap starting.
I20260927 16:34:20.812546 9671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:20.813429 9671 log.cc:824] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:20.815037 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:20.806928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.62:37069)
0927 16:34:20.807546 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:20.807566 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:20.807602 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:34:20.808245 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:20.808262 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:20.808657 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:20.808948 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:20.809293 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.810381 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.811553 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:20.812098 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:20.812228 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:20.813141 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:20.813146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:20.813182 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:20.813609 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:20.813689 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:20.813710 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:20.813788 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:20.813982 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 16:34:20.814299 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260927 16:34:20.815554 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3: No bootstrap required, opened a new log
I20260927 16:34:20.819239 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } }
I20260927 16:34:20.819489 9671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:20.819581 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cc9e8e5ebd4bfb9837debb6749a3a3, State: Initialized, Role: FOLLOWER
I20260927 16:34:20.820019 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } }
I20260927 16:34:20.820169 9671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:34:20.820257 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:34:20.820375 9671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:20.821339 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } }
I20260927 16:34:20.821702 9671 leader_election.cc:302] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cc9e8e5ebd4bfb9837debb6749a3a3; no voters:
I20260927 16:34:20.822011 9671 leader_election.cc:288] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:34:20.822196 9676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:20.822438 9676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 54cc9e8e5ebd4bfb9837debb6749a3a3, State: Running, Role: LEADER
I20260927 16:34:20.822824 9676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } }
I20260927 16:34:20.823410 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:34:20.824523 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cc9e8e5ebd4bfb9837debb6749a3a3. Latest consensus state: current_term: 1 leader_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } }
I20260927 16:34:20.824656 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:20.824932 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc9e8e5ebd4bfb9837debb6749a3a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37069 } } }
I20260927 16:34:20.825021 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:34:20.825037 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:20.828202 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:34:20.833537 9685 catalog_manager.cc:1347] Generated new cluster ID: 82a011df4f96477a92045e39cf7950cc
I20260927 16:34:20.833607 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:34:20.875967 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:34:20.876906 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:34:20.883203 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54cc9e8e5ebd4bfb9837debb6749a3a3: Generated new TSK 0
I20260927 16:34:20.883927 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:34:20.893733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 16:34:21.007758 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:21.008037 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:21.008075 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:21.012331 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:21.012405 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:21.012496 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:34:21.017427 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:21.018659 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:21.020115 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:21.030035 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.030011 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.030068 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.030103 9695 server_base.cc:1034] running on GCE node
I20260927 16:34:21.030773 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:21.031502 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:21.032856 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790526861032837 us; error 35 us; skew 500 ppm
I20260927 16:34:21.034946 9695 webserver.cc:460] Webserver started at http://127.0.98.1:34735/ using document root <none> and password file <none>
I20260927 16:34:21.035576 9695 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:21.035682 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:21.035930 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:21.038064 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ee51ff05d7049d9942e3c06287117c0"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "8b900634700c427f71b2df4077678900"
server_key_iv: "85c4b9b34f490a6e5018ded3016f73db"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.038807 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ee51ff05d7049d9942e3c06287117c0"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "8b900634700c427f71b2df4077678900"
server_key_iv: "85c4b9b34f490a6e5018ded3016f73db"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.042742 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:34:21.045424 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.046748 9695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:21.046888 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "0ee51ff05d7049d9942e3c06287117c0"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "8b900634700c427f71b2df4077678900"
server_key_iv: "85c4b9b34f490a6e5018ded3016f73db"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.047011 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:21.062865 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:21.063665 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:21.063894 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:21.064519 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:21.065510 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:21.065586 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.065660 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:21.065712 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.075278 9695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39453
I20260927 16:34:21.075320 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39453 every 8 connection(s)
I20260927 16:34:21.076270 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:34:21.080259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9695
I20260927 16:34:21.080394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:34:21.080701 392 external_mini_cluster.cc:1428] Setting key a1ba2c1e5a2668555b98f56a5d4da32a
I20260927 16:34:21.082844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:34:21.086809 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.078339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49681 (local address 127.0.98.62:37069)
0927 16:34:21.078480 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.078483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.079690 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 16:34:21.081271 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.081275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.081337 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.081468 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.082542 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.083375 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.084545 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.084937 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.085065 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.086196 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:21.086203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:21.086233 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:21.086252 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:21.086279 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:21.086294 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:21.086372 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:21.086540 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:34:21.086620 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:34:21.087636 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.078817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.98.1:49681)
0927 16:34:21.079530 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:21.079575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:21.081032 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:21.081629 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:21.081638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:21.081740 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:21.082383 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.082399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.083504 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.083507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.084417 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.084426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.085093 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.085096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.085243 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.085888 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:21.085889 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:21.085932 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:21.086003 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:21.086393 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:21.086400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:21.086404 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:34:21.086508 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:34:21.086745 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260927 16:34:21.089002 9823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37069
I20260927 16:34:21.089278 9823 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:21.089883 9823 heartbeater.cc:511] Master 127.0.98.62:37069 requested a full tablet report, sending...
I20260927 16:34:21.091424 9635 ts_manager.cc:194] Registered new tserver with Master: 0ee51ff05d7049d9942e3c06287117c0 (127.0.98.1:39453)
I20260927 16:34:21.092571 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49681
W20260927 16:34:21.196667 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:21.197016 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:21.197093 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:21.201197 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:21.201294 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:21.201388 9826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:34:21.206740 9826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:21.207851 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:21.209227 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:21.218791 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.218866 9826 server_base.cc:1034] running on GCE node
W20260927 16:34:21.218778 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.218849 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.219471 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:21.220117 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:21.221292 9826 hybrid_clock.cc:648] HybridClock initialized: now 1790526861221259 us; error 43 us; skew 500 ppm
I20260927 16:34:21.223281 9826 webserver.cc:460] Webserver started at http://127.0.98.2:37187/ using document root <none> and password file <none>
I20260927 16:34:21.223894 9826 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:21.223985 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:21.224211 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:21.226903 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "effe90088fe54d9fbc7423b6fc0a4abf"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "f51a230a860cc7669913f01544c61382"
server_key_iv: "bd92938256ebc2f3b17e860604d31820"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.227614 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "effe90088fe54d9fbc7423b6fc0a4abf"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "f51a230a860cc7669913f01544c61382"
server_key_iv: "bd92938256ebc2f3b17e860604d31820"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.231525 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:34:21.234211 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.235680 9826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:34:21.235800 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "effe90088fe54d9fbc7423b6fc0a4abf"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "f51a230a860cc7669913f01544c61382"
server_key_iv: "bd92938256ebc2f3b17e860604d31820"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.235891 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:21.254794 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:21.255442 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:21.255592 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:21.256119 9826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:21.257014 9826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:21.257066 9826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.257109 9826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:21.257129 9826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.265486 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44867
I20260927 16:34:21.265561 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44867 every 8 connection(s)
I20260927 16:34:21.266592 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:34:21.269942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9826
I20260927 16:34:21.270067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:34:21.270428 392 external_mini_cluster.cc:1428] Setting key df300920ac26ed4cb339da3f6eec39a8
I20260927 16:34:21.272575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:34:21.276469 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.268438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50751 (local address 127.0.98.62:37069)
0927 16:34:21.268575 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.268579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.269666 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 16:34:21.271473 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.271476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.271519 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.271592 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.272580 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.273400 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.274604 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.274830 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.274941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.276029 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:21.276032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:21.276034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:21.276046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:21.276064 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:21.276072 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:21.276133 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:21.276312 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:34:21.276361 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:34:21.277755 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.268851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.98.2:50751)
0927 16:34:21.269498 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:21.269537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:21.271247 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:21.271798 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:21.271809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:21.271924 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:21.272446 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.272459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.273530 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.273532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.274485 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.274494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.274981 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.274985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.275148 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.275731 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:21.275736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:21.275792 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:21.275874 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:21.276158 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:21.276169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:21.276177 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:34:21.276305 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 16:34:21.276604 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260927 16:34:21.278961 9954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37069
I20260927 16:34:21.279208 9954 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:21.279718 9954 heartbeater.cc:511] Master 127.0.98.62:37069 requested a full tablet report, sending...
I20260927 16:34:21.280786 9635 ts_manager.cc:194] Registered new tserver with Master: effe90088fe54d9fbc7423b6fc0a4abf (127.0.98.2:44867)
I20260927 16:34:21.281628 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50751
W20260927 16:34:21.384359 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:21.384691 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:21.384763 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:21.388782 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:21.388834 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:21.388890 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:34:21.393527 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:21.394591 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:21.395879 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:21.405508 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.405498 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.405546 9957 server_base.cc:1034] running on GCE node
W20260927 16:34:21.405498 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.406287 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:21.406879 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:21.408058 9957 hybrid_clock.cc:648] HybridClock initialized: now 1790526861408037 us; error 33 us; skew 500 ppm
I20260927 16:34:21.409762 9957 webserver.cc:460] Webserver started at http://127.0.98.3:44945/ using document root <none> and password file <none>
I20260927 16:34:21.410395 9957 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:21.410482 9957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:21.410717 9957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:21.412715 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "22c34533cfe3433abcdd66b7314a2e4e"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "a9c05f7d64a5f3b5595a2ea115c9d62e"
server_key_iv: "29bbb440cea26f30a20cc1077eb88f50"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.413261 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22c34533cfe3433abcdd66b7314a2e4e"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "a9c05f7d64a5f3b5595a2ea115c9d62e"
server_key_iv: "29bbb440cea26f30a20cc1077eb88f50"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.416826 9957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:34:21.419276 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.420400 9957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:34:21.420547 9957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "22c34533cfe3433abcdd66b7314a2e4e"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "a9c05f7d64a5f3b5595a2ea115c9d62e"
server_key_iv: "29bbb440cea26f30a20cc1077eb88f50"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.420655 9957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:21.441104 9957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:21.441834 9957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:21.442028 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:21.442665 9957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:21.443599 9957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:21.443668 9957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.443749 9957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:21.443804 9957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.452318 9957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38839
I20260927 16:34:21.452347 10084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38839 every 8 connection(s)
I20260927 16:34:21.453199 9957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:34:21.459654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 9957
I20260927 16:34:21.459748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:34:21.460000 392 external_mini_cluster.cc:1428] Setting key 83ea75574e8fd99f7370048b3fe3fc04
I20260927 16:34:21.462409 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.454972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39021 (local address 127.0.98.62:37069)
0927 16:34:21.455136 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.455141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.456174 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 16:34:21.457513 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.457515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.457560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.457633 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.458551 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.459078 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.460278 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.460565 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.460684 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.461909 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:21.461912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:21.461914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:21.461925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:21.461964 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:21.461976 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:21.462048 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:21.462237 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 16:34:21.462293 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:34:21.463246 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.455402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37069 (local address 127.0.98.3:39021)
0927 16:34:21.456029 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:21.456069 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:21.457334 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:21.457802 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:21.457808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:21.457891 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:21.458433 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.458445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.459179 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.459181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.460165 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:21.460174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:21.460739 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:21.460743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:21.460961 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:21.461618 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:21.461620 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:21.461670 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:21.461766 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:21.462063 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:21.462072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:21.462078 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:34:21.462224 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 16:34:21.462473 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260927 16:34:21.464279 10085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37069
I20260927 16:34:21.464478 10085 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:21.464932 10085 heartbeater.cc:511] Master 127.0.98.62:37069 requested a full tablet report, sending...
I20260927 16:34:21.465829 9635 ts_manager.cc:194] Registered new tserver with Master: 22c34533cfe3433abcdd66b7314a2e4e (127.0.98.3:38839)
I20260927 16:34:21.466416 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39021
I20260927 16:34:21.473839 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:34:21.476890 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:21.475759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.0.98.62:37069)
0927 16:34:21.475887 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.475891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.476144 (+ 253us) server_negotiation.cc:407] Connection header received
0927 16:34:21.476316 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.476319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.476397 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.476496 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.476756 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52438: BlockingRecv error: recv got EOF from 127.0.0.1:52438 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:34:21.477083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9695
I20260927 16:34:21.482415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9826
I20260927 16:34:21.487390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9957
I20260927 16:34:21.492367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 9604
2026-09-27T16:34:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (901 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T16:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:21Z Disabled control of system clock
I20260927 16:34:21.520861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42173
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:34:21.629106 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:21.629400 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:21.629439 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:21.633419 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:21.633481 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:21.633497 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:21.633514 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:21.633528 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:21.638527 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:21.639576 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:21.641009 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:34:21.649616 10096 server_base.cc:1034] running on GCE node
W20260927 16:34:21.649554 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.649554 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.649578 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.650440 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:21.651829 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:21.653002 10096 hybrid_clock.cc:648] HybridClock initialized: now 1790526861652980 us; error 43 us; skew 500 ppm
I20260927 16:34:21.655009 10096 webserver.cc:460] Webserver started at http://127.0.98.62:37745/ using document root <none> and password file <none>
I20260927 16:34:21.655642 10096 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:21.655745 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:21.655970 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:21.658020 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "7593e708aa8744ac8beb5a6346ccf7e9"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "0a639ecff175007ec12ad97415fd79a2"
server_key_iv: "998bf68ebfb4016cb3d3fa2a6981d841"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.658675 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "7593e708aa8744ac8beb5a6346ccf7e9"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "0a639ecff175007ec12ad97415fd79a2"
server_key_iv: "998bf68ebfb4016cb3d3fa2a6981d841"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.662418 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:34:21.664906 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.666270 10096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:21.666410 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "7593e708aa8744ac8beb5a6346ccf7e9"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "0a639ecff175007ec12ad97415fd79a2"
server_key_iv: "998bf68ebfb4016cb3d3fa2a6981d841"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.666534 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:21.671221 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:21.671855 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:21.672048 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:21.679127 10096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42173
I20260927 16:34:21.679148 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42173 every 8 connection(s)
I20260927 16:34:21.680099 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:34:21.683252 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:21.686347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10096
I20260927 16:34:21.686472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:34:21.686764 392 external_mini_cluster.cc:1428] Setting key 2049b4e5db5f2a54eb00f35e3fd75388
I20260927 16:34:21.689067 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: Bootstrap starting.
I20260927 16:34:21.691789 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:21.692222 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.688109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.98.62:42173)
0927 16:34:21.688722 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.688738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.688775 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:34:21.689452 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.689469 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.689825 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.690103 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.690931 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:21.690936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:21.690948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:21.690958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:21.691011 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:21.691026 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:21.691135 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:21.691223 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 16:34:21.691491 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260927 16:34:21.692572 10163 log.cc:824] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:21.694689 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: No bootstrap required, opened a new log
I20260927 16:34:21.698233 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 16:34:21.698483 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:21.698559 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7593e708aa8744ac8beb5a6346ccf7e9, State: Initialized, Role: FOLLOWER
I20260927 16:34:21.699024 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 16:34:21.699167 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:34:21.699259 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:34:21.699354 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:21.700245 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 16:34:21.700588 10163 leader_election.cc:302] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7593e708aa8744ac8beb5a6346ccf7e9; no voters:
I20260927 16:34:21.700883 10163 leader_election.cc:288] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:34:21.700999 10168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:21.701238 10168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7593e708aa8744ac8beb5a6346ccf7e9, State: Running, Role: LEADER
I20260927 16:34:21.701609 10168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } }
I20260927 16:34:21.702100 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:34:21.703210 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7593e708aa8744ac8beb5a6346ccf7e9. Latest consensus state: current_term: 1 leader_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } }
I20260927 16:34:21.703646 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:21.703540 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7593e708aa8744ac8beb5a6346ccf7e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42173 } } }
I20260927 16:34:21.704005 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9 [sys.catalog]: This master's current role is: LEADER
W20260927 16:34:21.706820 10183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:34:21.706909 10183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:34:21.707049 10181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:34:21.707899 10181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:34:21.712848 10181 catalog_manager.cc:1347] Generated new cluster ID: a6efd1b04da943af837a1299ccdd2fa9
I20260927 16:34:21.712913 10181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:34:21.743557 10181 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:34:21.743943 10181 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:34:21.754204 10181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7593e708aa8744ac8beb5a6346ccf7e9: Generated new TSK 0
I20260927 16:34:21.754944 10181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:34:21.757519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:34:21.871001 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:21.871310 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:21.871382 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:21.875600 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:21.875666 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:21.875725 10187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:34:21.880494 10187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:21.881536 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:21.882977 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:21.892596 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:21.892601 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.892664 10187 server_base.cc:1034] running on GCE node
W20260927 16:34:21.892601 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:21.893270 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:21.893890 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:21.895047 10187 hybrid_clock.cc:648] HybridClock initialized: now 1790526861895018 us; error 42 us; skew 500 ppm
I20260927 16:34:21.896973 10187 webserver.cc:460] Webserver started at http://127.0.98.1:40543/ using document root <none> and password file <none>
I20260927 16:34:21.897586 10187 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:21.897655 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:21.897900 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:21.899991 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "f54372e466c14d70b0f3bfbbe0afad20"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "b2829bf6ec4a7d85626987ed4da2e936"
server_key_iv: "377dc8add8bf2ad65dcbbdc6bc5d117b"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.900583 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f54372e466c14d70b0f3bfbbe0afad20"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "b2829bf6ec4a7d85626987ed4da2e936"
server_key_iv: "377dc8add8bf2ad65dcbbdc6bc5d117b"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.904445 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:34:21.906925 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.908164 10187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:21.908289 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "f54372e466c14d70b0f3bfbbe0afad20"
format_stamp: "Formatted at 2026-09-27 16:34:21 on dist-test-slave-t27c"
server_key: "b2829bf6ec4a7d85626987ed4da2e936"
server_key_iv: "377dc8add8bf2ad65dcbbdc6bc5d117b"
server_key_version: "encryptionkey@0"
I20260927 16:34:21.908401 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:21.913071 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:21.913723 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:21.913902 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:21.914505 10187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:21.915398 10187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:21.915444 10187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.915504 10187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:21.915536 10187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:21.923745 10187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42899
I20260927 16:34:21.923784 10314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42899 every 8 connection(s)
I20260927 16:34:21.924625 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:34:21.930574 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.926413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50397 (local address 127.0.98.62:42173)
0927 16:34:21.926545 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:21.926549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:21.927595 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 16:34:21.928913 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:21.928916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:21.928960 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:21.929067 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:21.930075 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:21.930079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:21.930081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:21.930093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:21.930115 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:21.930129 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:21.930251 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:21.930402 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 16:34:21.930446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:34:21.931373 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:21.926822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.1:50397)
0927 16:34:21.927456 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:21.927494 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:21.928709 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:21.929223 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:21.929230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:21.929312 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:21.929838 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:21.929839 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:21.929877 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:21.929938 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:21.930246 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:21.930254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:21.930263 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:34:21.930433 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 16:34:21.930652 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260927 16:34:21.932510 10315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 16:34:21.932729 10315 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:21.933099 10315 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 16:34:21.933877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10187
I20260927 16:34:21.934001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:34:21.934358 392 external_mini_cluster.cc:1428] Setting key 98a8b1dcc66057af4843adc76788c31c
I20260927 16:34:21.934669 10127 ts_manager.cc:194] Registered new tserver with Master: f54372e466c14d70b0f3bfbbe0afad20 (127.0.98.1:42899)
I20260927 16:34:21.936404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:34:22.045571 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:22.045849 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:22.045888 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:22.049940 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:22.050001 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:22.050092 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:34:22.054979 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:22.056143 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:22.057536 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:22.067396 10324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:22.067632 10318 server_base.cc:1034] running on GCE node
W20260927 16:34:22.067507 10323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:22.067396 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:22.068151 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:22.068794 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:22.069949 10318 hybrid_clock.cc:648] HybridClock initialized: now 1790526862069917 us; error 48 us; skew 500 ppm
I20260927 16:34:22.072073 10318 webserver.cc:460] Webserver started at http://127.0.98.2:44761/ using document root <none> and password file <none>
I20260927 16:34:22.072700 10318 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:22.072814 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:22.073043 10318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:22.075213 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "7847522d5cac4cb08303cc3933680a89"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "fa20d091b09fe5fa4ae50e363ccfdc5b"
server_key_iv: "0bfd7cfabc8e9cafe967ff6a4ca4e366"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.075798 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7847522d5cac4cb08303cc3933680a89"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "fa20d091b09fe5fa4ae50e363ccfdc5b"
server_key_iv: "0bfd7cfabc8e9cafe967ff6a4ca4e366"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.079450 10318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:34:22.081971 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.083285 10318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:22.083425 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "7847522d5cac4cb08303cc3933680a89"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "fa20d091b09fe5fa4ae50e363ccfdc5b"
server_key_iv: "0bfd7cfabc8e9cafe967ff6a4ca4e366"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.083539 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:22.088106 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:22.088712 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:22.088891 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:22.089414 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:22.090335 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:22.090402 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.090463 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:22.090512 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.099164 10318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37303
I20260927 16:34:22.099184 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37303 every 8 connection(s)
I20260927 16:34:22.100054 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:34:22.102079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10318
I20260927 16:34:22.102238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:34:22.102523 392 external_mini_cluster.cc:1428] Setting key d00afabb9ab5cfd060cf241c16e5f671
I20260927 16:34:22.105111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--builtin_ntp_servers=127.0.98.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 16:34:22.107398 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:22.101799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35937 (local address 127.0.98.62:42173)
0927 16:34:22.101977 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:22.101982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:22.103265 (+ 1283us) server_negotiation.cc:407] Connection header received
0927 16:34:22.105218 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:22.105227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:22.105278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:22.105381 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:22.106844 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:22.106850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:22.106854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:22.106869 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:22.106894 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:22.106908 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:22.107039 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:22.107184 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 16:34:22.107234 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:34:22.108224 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:22.102298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.2:35937)
0927 16:34:22.103080 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:22.103145 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:22.104919 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:22.105599 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:22.105613 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:22.105743 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:22.106479 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:22.106484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:22.106547 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:22.106650 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:22.107030 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:22.107038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:22.107044 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:34:22.107201 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 16:34:22.107392 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20260927 16:34:22.109475 10446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 16:34:22.109717 10446 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:22.110248 10446 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 16:34:22.111377 10127 ts_manager.cc:194] Registered new tserver with Master: 7847522d5cac4cb08303cc3933680a89 (127.0.98.2:37303)
W20260927 16:34:22.218576 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:22.218832 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:22.218868 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:22.222949 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:22.223001 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:22.223078 10449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:34:22.227799 10449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:22.228928 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:22.230773 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:22.240681 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:22.240703 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:22.240700 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:22.240769 10449 server_base.cc:1034] running on GCE node
I20260927 16:34:22.241297 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:22.241976 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:22.243180 10449 hybrid_clock.cc:648] HybridClock initialized: now 1790526862243142 us; error 54 us; skew 500 ppm
I20260927 16:34:22.244879 10449 webserver.cc:460] Webserver started at http://127.0.98.3:38499/ using document root <none> and password file <none>
I20260927 16:34:22.245473 10449 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:22.245566 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:22.245787 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:22.247944 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d96cdbb37f74fbb9f039ed038a80990"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "6b7f746b4f585fb87bb3a16216f13031"
server_key_iv: "f21a44b33b448e79d2aa3c6eda7cd65c"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.248515 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d96cdbb37f74fbb9f039ed038a80990"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "6b7f746b4f585fb87bb3a16216f13031"
server_key_iv: "f21a44b33b448e79d2aa3c6eda7cd65c"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.252117 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:34:22.254577 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.255940 10449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:34:22.256063 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "1d96cdbb37f74fbb9f039ed038a80990"
format_stamp: "Formatted at 2026-09-27 16:34:22 on dist-test-slave-t27c"
server_key: "6b7f746b4f585fb87bb3a16216f13031"
server_key_iv: "f21a44b33b448e79d2aa3c6eda7cd65c"
server_key_version: "encryptionkey@0"
I20260927 16:34:22.256165 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:22.260674 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:22.261258 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:22.261437 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:22.261981 10449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:22.262930 10449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:22.262976 10449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.263053 10449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:22.263108 10449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:22.271936 10449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38041
I20260927 16:34:22.271955 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38041 every 8 connection(s)
I20260927 16:34:22.272845 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:34:22.278672 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:22.274631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54083 (local address 127.0.98.62:42173)
0927 16:34:22.274784 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:22.274789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:22.275909 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 16:34:22.277123 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:22.277125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:22.277162 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:22.277238 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:22.278281 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:22.278284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:22.278286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:34:22.278297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:22.278315 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:34:22.278323 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:22.278398 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:22.278496 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 16:34:22.278533 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:34:22.279465 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:22.275109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42173 (local address 127.0.98.3:54083)
0927 16:34:22.275755 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:22.275794 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:22.276932 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:22.277366 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:22.277372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:22.277456 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:22.277980 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:34:22.277982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:22.278022 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:34:22.278090 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:22.278384 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:22.278389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:22.278392 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:34:22.278512 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 16:34:22.278706 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260927 16:34:22.280565 10577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42173
I20260927 16:34:22.280799 10577 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:22.281193 10577 heartbeater.cc:511] Master 127.0.98.62:42173 requested a full tablet report, sending...
I20260927 16:34:22.282223 10127 ts_manager.cc:194] Registered new tserver with Master: 1d96cdbb37f74fbb9f039ed038a80990 (127.0.98.3:38041)
I20260927 16:34:22.282577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10449
I20260927 16:34:22.282689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:34:22.283002 392 external_mini_cluster.cc:1428] Setting key 41555e416572759251998b483cdb1a1b
I20260927 16:34:22.284971 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:34:22.287742 10166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:34:22.286904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.98.62:42173)
0927 16:34:22.287018 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:22.287029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:22.287156 (+ 127us) server_negotiation.cc:407] Connection header received
0927 16:34:22.287249 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:22.287251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:22.287280 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:22.287398 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:22.287618 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:34:22.287884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10187
I20260927 16:34:22.293229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10318
I20260927 16:34:22.298437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10449
I20260927 16:34:22.303697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10096
2026-09-27T16:34:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (811 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:34:22 dist-test-slave-t27c krb5kdc[10587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:34:22 dist-test-slave-t27c krb5kdc[10587](info): set up 2 sockets
Sep 27 16:34:22 dist-test-slave-t27c krb5kdc[10587](info): commencing operation
krb5kdc: starting...
W20260927 16:34:23.347383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:34:23 dist-test-slave-t27c krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:34:23.471071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:23.579591 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:23.579860 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:23.579901 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:23.583773 10600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:34:23.583832 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:23.583853 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:23.583869 10600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:34:23.583884 10600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:34:23.589033 10600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:23.590037 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:23.591025 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:23.599422 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:23.599531 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:23.599618 10600 server_base.cc:1034] running on GCE node
W20260927 16:34:23.599531 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:23.600236 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:23.601342 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:23.602536 10600 hybrid_clock.cc:648] HybridClock initialized: now 1790526863602504 us; error 40 us; skew 500 ppm
Sep 27 16:34:23 dist-test-slave-t27c krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:23.610144 10600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:34:23.611150 10600 webserver.cc:460] Webserver started at http://127.0.98.62:43271/ using document root <none> and password file <none>
I20260927 16:34:23.611725 10600 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:23.611814 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:23.612059 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:23.614075 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "9981b76a89ba469ab6b4c764cdb6c30c"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "a20de42e214e50fef5fa545dedfd55ff"
server_key_iv: "5008a16f29bc61235b15113606da6b23"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.614818 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "9981b76a89ba469ab6b4c764cdb6c30c"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "a20de42e214e50fef5fa545dedfd55ff"
server_key_iv: "5008a16f29bc61235b15113606da6b23"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.618597 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:34:23.620877 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:23.622087 10600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:23.622239 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "9981b76a89ba469ab6b4c764cdb6c30c"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "a20de42e214e50fef5fa545dedfd55ff"
server_key_iv: "5008a16f29bc61235b15113606da6b23"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.622359 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:23.645757 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:23.648628 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:23.648833 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:23.655761 10600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36127
I20260927 16:34:23.656044 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36127 every 8 connection(s)
I20260927 16:34:23.656741 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:34:23.657050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10600
I20260927 16:34:23.657141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:34:23.657471 392 external_mini_cluster.cc:1428] Setting key 8827ce040b647ad4dfd07e77c7d77fd5
I20260927 16:34:23.659873 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 16:34:23 dist-test-slave-t27c krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:23.666834 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: Bootstrap starting.
I20260927 16:34:23.669708 10668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:23.670634 10668 log.cc:824] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:23.672389 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:23.658809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.0.98.62:36127)
0927 16:34:23.659212 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:23.659230 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:23.659276 (+ 46us) server_negotiation.cc:407] Connection header received
0927 16:34:23.659942 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:23.659967 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:23.660348 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:23.660644 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:23.661489 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.662574 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.663752 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.664220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.664337 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:23.667245 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:23.667252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:23.667268 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:23.667300 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:23.669162 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:23.669628 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:23.669636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:23.669645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:23.669788 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:23.670065 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:23.670068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:23.670070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:23.670644 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:23.670840 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:23.671130 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 16:34:23.671446 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260927 16:34:23.672900 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: No bootstrap required, opened a new log
I20260927 16:34:23.676769 10668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } }
I20260927 16:34:23.676999 10668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:23.677039 10668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9981b76a89ba469ab6b4c764cdb6c30c, State: Initialized, Role: FOLLOWER
I20260927 16:34:23.677454 10668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } }
I20260927 16:34:23.677572 10668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:34:23.677613 10668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:34:23.677677 10668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:23.678622 10668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } }
I20260927 16:34:23.678902 10668 leader_election.cc:302] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9981b76a89ba469ab6b4c764cdb6c30c; no voters:
I20260927 16:34:23.679250 10668 leader_election.cc:288] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:34:23.679366 10673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:23.679593 10673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [term 1 LEADER]: Becoming Leader. State: Replica: 9981b76a89ba469ab6b4c764cdb6c30c, State: Running, Role: LEADER
I20260927 16:34:23.679942 10673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } }
I20260927 16:34:23.680555 10668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:34:23.681631 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } } }
I20260927 16:34:23.681756 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:23.682042 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9981b76a89ba469ab6b4c764cdb6c30c. Latest consensus state: current_term: 1 leader_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9981b76a89ba469ab6b4c764cdb6c30c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36127 } } }
I20260927 16:34:23.682132 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c [sys.catalog]: This master's current role is: LEADER
I20260927 16:34:23.682128 10682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:34:23.685411 10682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:34:23.691057 10682 catalog_manager.cc:1347] Generated new cluster ID: 764ed8a15b174608b95648ee15df6116
I20260927 16:34:23.691154 10682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:34:23.702098 10682 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:34:23.703032 10682 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:34:23.720947 10682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: Generated new TSK 0
I20260927 16:34:23.721709 10682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:34:23.754858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:23.867554 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:23.867831 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:23.867875 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:23.872037 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:23.872104 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:23.872164 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:34:23.877414 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:23.878566 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:23.879575 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:23.889489 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:23.889453 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:23.889528 10694 server_base.cc:1034] running on GCE node
W20260927 16:34:23.889467 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:23.890126 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:23.890868 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:23.892025 10694 hybrid_clock.cc:648] HybridClock initialized: now 1790526863891993 us; error 45 us; skew 500 ppm
Sep 27 16:34:23 dist-test-slave-t27c krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:23.894728 10694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:34:23.895519 10694 webserver.cc:460] Webserver started at http://127.0.98.1:44157/ using document root <none> and password file <none>
I20260927 16:34:23.896073 10694 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:23.896135 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:23.896292 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:23.898281 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "a75a9dc5fae54706a6068c7feed023aa"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "06cf63ba8784dc64eedde37b8d01785b"
server_key_iv: "8968d4d13bd55d2d7cd14b0730eebcf2"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.898804 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a75a9dc5fae54706a6068c7feed023aa"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "06cf63ba8784dc64eedde37b8d01785b"
server_key_iv: "8968d4d13bd55d2d7cd14b0730eebcf2"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.902520 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 16:34:23.904775 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:23.906004 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:34:23.906126 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "a75a9dc5fae54706a6068c7feed023aa"
format_stamp: "Formatted at 2026-09-27 16:34:23 on dist-test-slave-t27c"
server_key: "06cf63ba8784dc64eedde37b8d01785b"
server_key_iv: "8968d4d13bd55d2d7cd14b0730eebcf2"
server_key_version: "encryptionkey@0"
I20260927 16:34:23.906266 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:23.927749 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:23.930938 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:23.931160 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:23.931785 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:23.932618 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:23.932668 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:23.932708 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:23.932763 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:23.941792 10694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37291
I20260927 16:34:23.941809 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37291 every 8 connection(s)
I20260927 16:34:23.942782 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:34:23.951397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10694
I20260927 16:34:23.951498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:34:23.951751 392 external_mini_cluster.cc:1428] Setting key 2ce54990adaef64ec4f7c951a72b5271
Sep 27 16:34:23 dist-test-slave-t27c krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:23.957474 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:23.944634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56241 (local address 127.0.98.62:36127)
0927 16:34:23.944797 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:23.944800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:23.945864 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 16:34:23.946902 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:23.946906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:23.946963 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:23.947066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:23.948320 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.948918 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.950142 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.950432 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.950565 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:23.953460 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:23.953463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:23.953466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:23.953495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:23.955216 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:23.955727 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:23.955733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:23.955737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:23.955837 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:23.956167 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:23.956173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:23.956178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:23.956448 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:23.956575 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:23.957053 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 16:34:23.957207 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:34:23.958108 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:23.945030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36127 (local address 127.0.98.1:56241)
0927 16:34:23.945709 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:23.945752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:23.946679 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:23.947245 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:23.947252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:23.947648 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:23.948157 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:23.948170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.949059 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.949063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:23.950009 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:23.950018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:23.950603 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:23.950608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:23.950785 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:23.951486 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:23.951514 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:23.953284 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:23.955343 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:23.955348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:23.955360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:23.955625 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:23.955932 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:23.955934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:23.955936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:23.956072 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:23.956566 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:23.956571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:23.956803 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 16:34:23.956959 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:34:23.957219 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":112,"threads_started":1}
I20260927 16:34:23.959574 10823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36127
I20260927 16:34:23.959820 10823 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:23.960321 10823 heartbeater.cc:511] Master 127.0.98.62:36127 requested a full tablet report, sending...
I20260927 16:34:23.962087 10632 ts_manager.cc:194] Registered new tserver with Master: a75a9dc5fae54706a6068c7feed023aa (127.0.98.1:37291)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:34:23.963563 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:34:23.981336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:24.092589 10828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:24.092900 10828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:24.092976 10828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:24.097177 10828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:24.097262 10828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:24.097352 10828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:34:24.102883 10828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:24.103948 10828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:24.105022 10828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:34:24.114603 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:24.114676 10833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:24.114610 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:24.114900 10828 server_base.cc:1034] running on GCE node
I20260927 16:34:24.115355 10828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:24.116012 10828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:24.117218 10828 hybrid_clock.cc:648] HybridClock initialized: now 1790526864117198 us; error 39 us; skew 500 ppm
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:24.120033 10828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:34:24.120961 10828 webserver.cc:460] Webserver started at http://127.0.98.2:44307/ using document root <none> and password file <none>
I20260927 16:34:24.121516 10828 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:24.121574 10828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:24.121807 10828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:24.123929 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "027c0dd91ade4feea46f0776194b29d7"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "e71a2dc0a8d4abcaf172ae9d39e3888f"
server_key_iv: "08471d7a4772aa4d6aaf5afdb636a938"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.124495 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "027c0dd91ade4feea46f0776194b29d7"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "e71a2dc0a8d4abcaf172ae9d39e3888f"
server_key_iv: "08471d7a4772aa4d6aaf5afdb636a938"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.128352 10828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 16:34:24.130704 10843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.131958 10828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:34:24.132193 10828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "027c0dd91ade4feea46f0776194b29d7"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "e71a2dc0a8d4abcaf172ae9d39e3888f"
server_key_iv: "08471d7a4772aa4d6aaf5afdb636a938"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.132316 10828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:24.159804 10828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:24.163484 10828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:24.163728 10828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:24.164361 10828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:24.165215 10828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:24.165287 10828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.165359 10828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:24.165409 10828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.174556 10828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42889
I20260927 16:34:24.174577 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42889 every 8 connection(s)
I20260927 16:34:24.175678 10828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:34:24.178249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10828
I20260927 16:34:24.178359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:34:24.178617 392 external_mini_cluster.cc:1428] Setting key cd3007ea82fe81e0db5884b713c9a2a5
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:24.196604 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.180367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50907 (local address 127.0.98.62:36127)
0927 16:34:24.180483 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.180488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.182453 (+ 1965us) server_negotiation.cc:407] Connection header received
0927 16:34:24.183512 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.183514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.183559 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.183627 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:24.185220 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.185732 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.186960 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.187201 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.187315 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.192353 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:24.192356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:24.192359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:24.192384 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:24.194210 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.194744 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.194750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.194755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.194856 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.195275 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.195280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.195284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.195557 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:24.195668 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.196188 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.196338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:34:24.197356 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.180943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36127 (local address 127.0.98.2:50907)
0927 16:34:24.182309 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.182354 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.183280 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.183804 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.183815 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.184343 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:24.185071 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.185092 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.185861 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.185865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.186835 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.186844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.187350 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.187354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.187519 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.188124 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:24.188141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:24.192153 (+ 4012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:24.194328 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:24.194333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:24.194347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:24.194619 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:24.194948 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:24.194950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:24.194958 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:24.195153 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:24.195660 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:24.195665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:24.195904 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.196073 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.196392 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":141,"threads_started":1}
I20260927 16:34:24.198980 10957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36127
I20260927 16:34:24.199245 10957 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:24.199750 10957 heartbeater.cc:511] Master 127.0.98.62:36127 requested a full tablet report, sending...
I20260927 16:34:24.200832 10632 ts_manager.cc:194] Registered new tserver with Master: 027c0dd91ade4feea46f0776194b29d7 (127.0.98.2:42889)
I20260927 16:34:24.201489 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:34:24.208854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:34:24.319972 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:34:24.320250 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:34:24.320358 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:34:24.324430 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:34:24.324510 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:34:24.324594 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:34:24.329852 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:34:24.330986 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:34:24.332018 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:34:24.341619 10962 server_base.cc:1034] running on GCE node
W20260927 16:34:24.341593 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:24.341553 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:34:24.341557 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:34:24.342298 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:34:24.342923 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:34:24.344094 10962 hybrid_clock.cc:648] HybridClock initialized: now 1790526864344079 us; error 30 us; skew 500 ppm
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:34:24.346853 10962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:34:24.347678 10962 webserver.cc:460] Webserver started at http://127.0.98.3:45633/ using document root <none> and password file <none>
I20260927 16:34:24.348232 10962 fs_manager.cc:359] Metadata directory not provided
I20260927 16:34:24.348317 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:34:24.348557 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:34:24.350606 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3f20bded6f04cc180a9fbd9525d43f4"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "4289efd9ea9a710d1626d1c68eb45760"
server_key_iv: "6b4a2f72a793eedc9d3c8dfe759fb182"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.351138 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3f20bded6f04cc180a9fbd9525d43f4"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "4289efd9ea9a710d1626d1c68eb45760"
server_key_iv: "6b4a2f72a793eedc9d3c8dfe759fb182"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.354708 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:34:24.356971 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.358240 10962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:34:24.358369 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "d3f20bded6f04cc180a9fbd9525d43f4"
format_stamp: "Formatted at 2026-09-27 16:34:24 on dist-test-slave-t27c"
server_key: "4289efd9ea9a710d1626d1c68eb45760"
server_key_iv: "6b4a2f72a793eedc9d3c8dfe759fb182"
server_key_version: "encryptionkey@0"
I20260927 16:34:24.358480 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:34:24.369144 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:34:24.371857 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:34:24.372054 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:34:24.372603 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:34:24.373386 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:34:24.373461 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.373549 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:34:24.373601 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:34:24.382378 10962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45619
I20260927 16:34:24.382457 11090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45619 every 8 connection(s)
I20260927 16:34:24.383322 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:34:24.384824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 10962
I20260927 16:34:24.384934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:34:24.385215 392 external_mini_cluster.cc:1428] Setting key 68a3c5f3c0b05b273c0cfbeca49e7d4a
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:34:24.396822 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.386407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57015 (local address 127.0.98.62:36127)
0927 16:34:24.386533 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.386537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.386773 (+ 236us) server_negotiation.cc:407] Connection header received
0927 16:34:24.387771 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.387773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.387815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.387880 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:24.389044 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.389555 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.390768 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.391014 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.391095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.393458 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:24.393462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:24.393464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:24.393488 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:24.395011 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.395465 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.395468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.395469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.395541 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.395824 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.395826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.395827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.396004 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:24.396075 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.396511 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.396649 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:34:24.397578 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.385554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36127 (local address 127.0.98.3:57015)
0927 16:34:24.386606 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.386649 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.387587 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.388040 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.388047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.388434 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:34:24.388912 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.388925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.389657 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.389659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.390649 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.390659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.391145 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.391147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.391286 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.391820 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:34:24.391836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:34:24.393286 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:34:24.395108 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:24.395113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:24.395125 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:24.395373 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:24.395618 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:34:24.395620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:34:24.395621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:34:24.395747 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:34:24.396086 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:34:24.396091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:34:24.396315 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.396468 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.396722 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":672,"threads_started":1}
I20260927 16:34:24.398877 11091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36127
I20260927 16:34:24.399138 11091 heartbeater.cc:464] Registering TS with master...
I20260927 16:34:24.399695 11091 heartbeater.cc:511] Master 127.0.98.62:36127 requested a full tablet report, sending...
I20260927 16:34:24.400642 10631 ts_manager.cc:194] Registered new tserver with Master: d3f20bded6f04cc180a9fbd9525d43f4 (127.0.98.3:45619)
I20260927 16:34:24.401230 10631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57015
I20260927 16:34:24.411356 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:34:24.420132 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.413325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.0.98.62:36127)
0927 16:34:24.413455 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.413460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.413672 (+ 212us) server_negotiation.cc:407] Connection header received
0927 16:34:24.413772 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.413775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.413816 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.413892 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:34:24.414628 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.415136 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.415919 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.416127 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.416197 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.417421 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:34:24.417424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:34:24.417426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:34:24.417447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:34:24.418831 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.419175 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.419178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.419179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.419246 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:34:24.419461 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:34:24.419463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:34:24.419464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:34:24.419655 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:34:24.419752 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.419920 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.420015 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:34:24.422930 10631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:34:24.425988 10631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:34:24.440384 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.434962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42889 (local address 127.0.0.1:46358)
0927 16:34:24.435560 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.435574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.435895 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.436339 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.436342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.436359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:24.436473 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.436483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437504 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.437510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.438946 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.438955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.439907 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.439910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.440007 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.440037 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.440121 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.440204 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":161,"thread_start_us":95,"threads_started":1}
I20260927 16:34:24.440455 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.435191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.2:42889)
0927 16:34:24.435812 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.435821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.435839 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:34:24.436052 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.436058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.436206 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.436354 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:24.436627 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437383 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.439065 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.439787 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.439892 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.440042 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.440176 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.440302 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":104,"threads_started":1}
I20260927 16:34:24.440857 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.434451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.0.98.3:45619)
0927 16:34:24.434972 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.434983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.435186 (+ 203us) server_negotiation.cc:407] Connection header received
0927 16:34:24.435496 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.435501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.435695 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.435827 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:24.436533 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437703 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.439369 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.440197 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.440316 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.440443 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.440615 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.440710 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":139,"threads_started":1}
I20260927 16:34:24.441314 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.434524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45619 (local address 127.0.0.1:49026)
0927 16:34:24.435086 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.435101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.435314 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.436063 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.436069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.436102 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:24.436391 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.436401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437853 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.437856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.439243 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.439254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.440325 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.440328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.440418 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.440460 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.441065 (+ 605us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.441174 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":222,"spinlock_wait_cycles":10368,"thread_start_us":95,"threads_started":1}
I20260927 16:34:24.442296 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.434242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.0.1:53388)
0927 16:34:24.435052 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.435101 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.435299 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.436063 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.436069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.436100 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:24.436352 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.436361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437853 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.437856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.439246 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.439253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.441865 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.441869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.441970 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.441992 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.442074 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.442135 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":182,"spinlock_wait_cycles":4096,"thread_start_us":72,"threads_started":1}
I20260927 16:34:24.442358 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.434390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.1:37291)
0927 16:34:24.434785 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.434801 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.435187 (+ 386us) server_negotiation.cc:407] Connection header received
0927 16:34:24.435482 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.435488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.435695 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.435827 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:24.436494 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.437700 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.440728 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.441761 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.441871 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.442007 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.442099 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.442228 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":108,"threads_started":1}
I20260927 16:34:24.443786 10891 tablet_service.cc:1467] Processing CreateTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 (DEFAULT_TABLE table=test-table [id=b26114cd752c4764b5cfd5317b0a4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:24.444257 11025 tablet_service.cc:1467] Processing CreateTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 (DEFAULT_TABLE table=test-table [id=b26114cd752c4764b5cfd5317b0a4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:24.444967 10891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a8143b19ff48e7b58b266032a43ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:24.445255 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a8143b19ff48e7b58b266032a43ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:24.445310 10757 tablet_service.cc:1467] Processing CreateTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 (DEFAULT_TABLE table=test-table [id=b26114cd752c4764b5cfd5317b0a4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:34:24.446274 10757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a8143b19ff48e7b58b266032a43ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:34:24.451218 11108 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Bootstrap starting.
I20260927 16:34:24.452787 11110 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: Bootstrap starting.
I20260927 16:34:24.453378 11109 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Bootstrap starting.
I20260927 16:34:24.453995 11108 tablet_bootstrap.cc:654] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:24.454879 11108 log.cc:824] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:24.455006 11110 tablet_bootstrap.cc:654] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:24.455571 11109 tablet_bootstrap.cc:654] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Neither blocks nor log segments found. Creating new log.
I20260927 16:34:24.456269 11110 log.cc:824] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:24.456426 11109 log.cc:824] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:34:24.457329 11108 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: No bootstrap required, opened a new log
I20260927 16:34:24.457520 11108 ts_tablet_manager.cc:1397] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:34:24.458206 11110 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: No bootstrap required, opened a new log
I20260927 16:34:24.458376 11110 ts_tablet_manager.cc:1397] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:34:24.459453 11109 tablet_bootstrap.cc:492] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: No bootstrap required, opened a new log
I20260927 16:34:24.459651 11109 ts_tablet_manager.cc:1397] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:34:24.461491 11108 raft_consensus.cc:348] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.461814 11108 raft_consensus.cc:374] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:24.461918 11108 raft_consensus.cc:729] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f20bded6f04cc180a9fbd9525d43f4, State: Initialized, Role: FOLLOWER
I20260927 16:34:24.462180 11110 raft_consensus.cc:348] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.462424 11110 raft_consensus.cc:374] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:24.462428 11108 consensus_queue.cc:260] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.462523 11110 raft_consensus.cc:729] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75a9dc5fae54706a6068c7feed023aa, State: Initialized, Role: FOLLOWER
I20260927 16:34:24.462983 11109 raft_consensus.cc:348] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.463132 11110 consensus_queue.cc:260] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.463295 11109 raft_consensus.cc:374] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:34:24.463372 11109 raft_consensus.cc:729] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027c0dd91ade4feea46f0776194b29d7, State: Initialized, Role: FOLLOWER
I20260927 16:34:24.463543 11108 ts_tablet_manager.cc:1428] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:34:24.463804 11109 consensus_queue.cc:260] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.463898 11091 heartbeater.cc:503] Master 127.0.98.62:36127 was elected leader, sending a full tablet report...
I20260927 16:34:24.463989 11110 ts_tablet_manager.cc:1428] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:34:24.464279 10823 heartbeater.cc:503] Master 127.0.98.62:36127 was elected leader, sending a full tablet report...
I20260927 16:34:24.464630 11109 ts_tablet_manager.cc:1428] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:34:24.465137 10957 heartbeater.cc:503] Master 127.0.98.62:36127 was elected leader, sending a full tablet report...
I20260927 16:34:24.570467 11116 raft_consensus.cc:482] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:34:24.570701 11116 raft_consensus.cc:504] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.571936 11116 leader_election.cc:288] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a75a9dc5fae54706a6068c7feed023aa (127.0.98.1:37291), d3f20bded6f04cc180a9fbd9525d43f4 (127.0.98.3:45619)
I20260927 16:34:24.577037 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.572250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49689 (local address 127.0.98.1:37291)
0927 16:34:24.572366 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.572372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.572513 (+ 141us) server_negotiation.cc:407] Connection header received
0927 16:34:24.572581 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.572584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.572658 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.572746 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:24.573070 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.573869 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.575739 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.576355 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.576486 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.576542 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.576834 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.576902 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:34:24.577035 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.572301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58547 (local address 127.0.98.3:45619)
0927 16:34:24.572458 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.572463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.572662 (+ 199us) server_negotiation.cc:407] Connection header received
0927 16:34:24.572724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.572730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.572782 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.572856 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:34:24.573106 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.573870 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.575739 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.576357 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.576478 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.576543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.576834 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.576902 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:34:24.577062 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.572224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45619 (local address 127.0.98.2:58547)
0927 16:34:24.572557 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.572570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.572649 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.572879 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.572882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.572899 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:24.572996 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.573006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.573998 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.574001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.575586 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.575597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.576492 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.576495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.576650 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.576722 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.576800 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.576896 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":222,"thread_start_us":136,"threads_started":1}
I20260927 16:34:24.577062 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.572173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.98.2:49689)
0927 16:34:24.572364 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.572380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:34:24.572479 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:34:24.572774 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:34:24.572776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:34:24.572788 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:34:24.572925 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.572932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.573998 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.574001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.575586 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:34:24.575597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.576492 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.576495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:34:24.576668 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.576725 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 16:34:24.576800 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:34:24.576896 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":231}
I20260927 16:34:24.577857 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a8143b19ff48e7b58b266032a43ee2" candidate_uuid: "027c0dd91ade4feea46f0776194b29d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75a9dc5fae54706a6068c7feed023aa" is_pre_election: true
I20260927 16:34:24.577859 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a8143b19ff48e7b58b266032a43ee2" candidate_uuid: "027c0dd91ade4feea46f0776194b29d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" is_pre_election: true
I20260927 16:34:24.578200 11045 raft_consensus.cc:2454] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027c0dd91ade4feea46f0776194b29d7 in term 0.
I20260927 16:34:24.578200 10777 raft_consensus.cc:2454] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027c0dd91ade4feea46f0776194b29d7 in term 0.
I20260927 16:34:24.578709 10846 leader_election.cc:302] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 027c0dd91ade4feea46f0776194b29d7, a75a9dc5fae54706a6068c7feed023aa, d3f20bded6f04cc180a9fbd9525d43f4; no voters:
I20260927 16:34:24.578949 11116 raft_consensus.cc:2781] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:34:24.579028 11116 raft_consensus.cc:482] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:34:24.579070 11116 raft_consensus.cc:3037] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:24.580070 11116 raft_consensus.cc:504] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.580399 11116 leader_election.cc:288] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [CANDIDATE]: Term 1 election: Requested vote from peers a75a9dc5fae54706a6068c7feed023aa (127.0.98.1:37291), d3f20bded6f04cc180a9fbd9525d43f4 (127.0.98.3:45619)
I20260927 16:34:24.580745 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a8143b19ff48e7b58b266032a43ee2" candidate_uuid: "027c0dd91ade4feea46f0776194b29d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75a9dc5fae54706a6068c7feed023aa"
I20260927 16:34:24.580812 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a8143b19ff48e7b58b266032a43ee2" candidate_uuid: "027c0dd91ade4feea46f0776194b29d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f20bded6f04cc180a9fbd9525d43f4"
I20260927 16:34:24.580859 10777 raft_consensus.cc:3037] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:24.580973 11045 raft_consensus.cc:3037] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:34:24.582086 10777 raft_consensus.cc:2454] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027c0dd91ade4feea46f0776194b29d7 in term 1.
I20260927 16:34:24.582237 11045 raft_consensus.cc:2454] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027c0dd91ade4feea46f0776194b29d7 in term 1.
I20260927 16:34:24.582412 10845 leader_election.cc:302] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027c0dd91ade4feea46f0776194b29d7, a75a9dc5fae54706a6068c7feed023aa; no voters:
I20260927 16:34:24.582620 11116 raft_consensus.cc:2781] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:34:24.582846 11116 raft_consensus.cc:686] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 1 LEADER]: Becoming Leader. State: Replica: 027c0dd91ade4feea46f0776194b29d7, State: Running, Role: LEADER
I20260927 16:34:24.583163 11116 consensus_queue.cc:237] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } }
I20260927 16:34:24.586328 10632 catalog_manager.cc:5495] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 027c0dd91ade4feea46f0776194b29d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "027c0dd91ade4feea46f0776194b29d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "027c0dd91ade4feea46f0776194b29d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42889 } health_report { overall_health: HEALTHY } } }
I20260927 16:34:24.628356 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.624986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.98.2:42889)
0927 16:34:24.625121 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:34:24.625124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:34:24.625137 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:34:24.625230 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:34:24.625232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:34:24.625286 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:34:24.625386 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:34:24.625612 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.626188 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.627097 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:34:24.627300 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:34:24.627377 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:34:24.627628 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:34:24.628048 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:34:24.628135 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:34:24.628185 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 16:34:24.628230 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:34:24.634135 11045 raft_consensus.cc:1261] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 1 FOLLOWER]: Refusing update from remote peer 027c0dd91ade4feea46f0776194b29d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 16:34:24.634261 11092 tablet.cc:2369] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:34:24.634327 10777 raft_consensus.cc:1261] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 1 FOLLOWER]: Refusing update from remote peer 027c0dd91ade4feea46f0776194b29d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:34:24.635047 11116 consensus_queue.cc:1035] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a75a9dc5fae54706a6068c7feed023aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:24.635267 11118 consensus_queue.cc:1035] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f20bded6f04cc180a9fbd9525d43f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:34:24.644748 11122 mvcc.cc:205] Tried to move back new op lower bound from 7333998037534609408 to 7333998037335552000. Current Snapshot: MvccSnapshot[applied={T|T < 7333998037534609408 or (T in {7333998037534609408})}]
I20260927 16:34:24.650633 10632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:24.650808 10632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:34:24.653100 10632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9981b76a89ba469ab6b4c764cdb6c30c: Sending DeleteTablet for 3 replicas of tablet 60a8143b19ff48e7b58b266032a43ee2
I20260927 16:34:24.653831 10757 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53388
I20260927 16:34:24.653885 11025 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49026
I20260927 16:34:24.654053 10891 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a8143b19ff48e7b58b266032a43ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:34:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260927 16:34:24.654386 11131 tablet_replica.cc:330] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa: stopping tablet replica
I20260927 16:34:24.654670 11132 tablet_replica.cc:330] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: stopping tablet replica
I20260927 16:34:24.654876 11131 raft_consensus.cc:2229] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:24.654942 11132 raft_consensus.cc:2229] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:34:24.655169 11131 raft_consensus.cc:2258] T 60a8143b19ff48e7b58b266032a43ee2 P a75a9dc5fae54706a6068c7feed023aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:24.655231 11132 raft_consensus.cc:2258] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:24.655285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10694
I20260927 16:34:24.659641 11132 ts_tablet_manager.cc:1905] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:24.662025 11133 tablet_replica.cc:330] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: stopping tablet replica
I20260927 16:34:24.662374 11133 raft_consensus.cc:2229] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:34:24.662814 11133 raft_consensus.cc:2258] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:34:24.663703 11132 ts_tablet_manager.cc:1918] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:34:24.663805 11132 log.cc:1196] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/60a8143b19ff48e7b58b266032a43ee2
I20260927 16:34:24.664127 11132 ts_tablet_manager.cc:1939] T 60a8143b19ff48e7b58b266032a43ee2 P d3f20bded6f04cc180a9fbd9525d43f4: Deleting consensus metadata
I20260927 16:34:24.665419 10618 catalog_manager.cc:4841] TS d3f20bded6f04cc180a9fbd9525d43f4 (127.0.98.3:45619): tablet 60a8143b19ff48e7b58b266032a43ee2 (table test-table [id=b26114cd752c4764b5cfd5317b0a4009]) successfully deleted
W20260927 16:34:24.666087 10617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37291 (error 108)
I20260927 16:34:24.666831 11133 ts_tablet_manager.cc:1905] T 60a8143b19ff48e7b58b266032a43ee2 P 027c0dd91ade4feea46f0776194b29d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:34:24.666932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10828
I20260927 16:34:24.669021 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.668707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37291 (local address 127.0.0.1:53392)
0927 16:34:24.668870 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.668941 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 16:34:24.669198 10617 catalog_manager.cc:4584] TS a75a9dc5fae54706a6068c7feed023aa (127.0.98.1:37291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60a8143b19ff48e7b58b266032a43ee2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37291: connect: Connection refused (error 111)
I20260927 16:34:24.672468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10962
I20260927 16:34:24.674381 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:34:24.672634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42889 (local address 127.0.0.1:46382)
0927 16:34:24.674221 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0927 16:34:24.674288 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1458}
I20260927 16:34:24.678232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 10600
2026-09-27T16:34:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2376 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[11137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[11137](info): set up 2 sockets
Sep 27 16:34:24 dist-test-slave-t27c krb5kdc[11137](info): commencing operation
krb5kdc: starting...
W20260927 16:34:25.811134 392 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:34:25 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:34:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 16:34:25.989156 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxe3jhj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/postgres -l logfile start
2026-09-27 16:34:28.516 UTC [11167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:34:28.516 UTC [11167] LOG: listening on IPv4 address "127.0.98.20", port 48077
2026-09-27 16:34:28.519 UTC [11167] LOG: listening on Unix socket "/tmp/.s.PGSQL.48077"
2026-09-27 16:34:28.554 UTC [11170] LOG: database system was shut down at 2026-09-27 16:34:27 UTC
2026-09-27 16:34:28.559 UTC [11167] LOG: database system is ready to accept connections
I20260927 16:34:29.507040 392 mini_postgres.cc:96] Postgres bound to 48077
2026-09-27 16:34:29.513 UTC [11178] FATAL: database "slave" does not exist
127.0.98.20:48077 - accepting connections
I20260927 16:34:29.514210 392 mini_ranger.cc:161] Starting Ranger...
I20260927 16:34:29.529281 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:34:30.020220 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:34:30.020323 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin
2026-09-27 16:34:30,299 [I] DB FLAVOR :POSTGRES
2026-09-27 16:34:30,299 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:34:30,300 [I] Checking connection..
2026-09-27 16:34:30,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:34:30,559 [I] Checking connection passed.
2026-09-27 16:34:30,559 [I] --------- Verifying version history table ---------
2026-09-27 16:34:30,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:34:30,823 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:34:30,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:34:31,057 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:34:31,057 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:34:31,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:34:31.284 UTC [11242] WARNING: there is no transaction in progress
2026-09-27 16:34:31,290 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:34:31,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:34:31,519 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:34:31,520 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:34:31,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:34:31,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:34:32,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t27c.c.gcp-upstream.internal','N') ;"
2026-09-27 16:34:32,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:34:32,492 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:34:32,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:34:32,766 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:34:32,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:34:33,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:34:33,259 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:34:33,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:34:34.242 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.257 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.272 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.288 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.303 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.321 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.386 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.392 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.406 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.415 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.425 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.438 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.447 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.453 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.459 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:34.466 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 16:34:35,074 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:34:35,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:34:35,313 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:34:35,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:34:35,543 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:34:35,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:34:35,803 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:34:35,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:34:36,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:34:36,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t27c.c.gcp-upstream.internal';"
2026-09-27 16:34:36,547 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:34:36,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:34:36,780 [I] DB_PATCHES have already been applied
I20260927 16:34:36.786096 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxe3jhj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:34:36.786218 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 16:34:36.789625 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50137
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:34:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:34:37.907050 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.393s user 0.000s sys 0.010s
Sep 27, 2026 4:34:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:34:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 4:34:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:34:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:34:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:34:59.690913 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:34:59 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526899, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:34:59.796208 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:34:59.807128 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:35:00.693634 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:35:00.693737 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms
I20260927 16:35:00.693843 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48077
2026-09-27 16:35:00,880 [I] DB FLAVOR :POSTGRES
2026-09-27 16:35:00,881 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:35:00,881 [I] Checking connection
2026-09-27 16:35:00,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:35:01,126 [I] connection success
2026-09-27 16:35:01,126 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:35:01,126 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:35:01,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:35:01,381 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:35:01,382 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:35:01,382 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:35:01,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:35:01,647 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:35:02.100427 392 mini_ranger_kms.cc:310] Created kms service
I20260927 16:35:02.320117 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:35:02.364359 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:35:02 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526899, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:02.749680 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:35:02.749858 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:35:02.749890 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 16:35:02.754243 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11530
I20260927 16:35:02.754379 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51031
Sep 27, 2026 4:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:35:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:35:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 16:35:05.025786 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.230s user 0.002s sys 0.168s
I20260927 16:35:05.025926 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:35:05.026046 392 mini_ranger_kms.cc:401] 127.0.98.20:51031/kms/v1/keys
Sep 27 16:35:05 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 16:35:05 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:35:05 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526905, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:35:08 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:35:08.279091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41551
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51031/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:08.412634 11581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:08.412925 11581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:08.412966 11581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:08.417238 11581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:35:08.417316 11581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:08.417337 11581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:08.417354 11581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:35:08.417395 11581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:35:08.423460 11581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41551
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:08.424638 11581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:08.425761 11581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:08.434338 11587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:08.434347 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:08.434338 11589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:08.434713 11581 server_base.cc:1034] running on GCE node
I20260927 16:35:08.435365 11581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:08.436605 11581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:08.437783 11581 hybrid_clock.cc:648] HybridClock initialized: now 1790526908437762 us; error 47 us; skew 500 ppm
Sep 27 16:35:08 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:08.440721 11581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:35:08.441733 11581 webserver.cc:460] Webserver started at http://127.0.98.62:35653/ using document root <none> and password file <none>
I20260927 16:35:08.442341 11581 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:08.442401 11581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:08.442629 11581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:08 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:08.502503 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "292daa1fa46544fcbd7c3f94c344737c"
format_stamp: "Formatted at 2026-09-27 16:35:08 on dist-test-slave-t27c"
server_key: "ba4722a41d00425a086feff97f603908"
server_key_iv: "61f66eb6855b0b64c2376548da8a4b59"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:08.503156 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "292daa1fa46544fcbd7c3f94c344737c"
format_stamp: "Formatted at 2026-09-27 16:35:08 on dist-test-slave-t27c"
server_key: "ba4722a41d00425a086feff97f603908"
server_key_iv: "61f66eb6855b0b64c2376548da8a4b59"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:08.507136 11581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:35:08.524995 11596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:08.526520 11581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:35:08.526688 11581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "292daa1fa46544fcbd7c3f94c344737c"
format_stamp: "Formatted at 2026-09-27 16:35:08 on dist-test-slave-t27c"
server_key: "ba4722a41d00425a086feff97f603908"
server_key_iv: "61f66eb6855b0b64c2376548da8a4b59"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:08.526824 11581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:08.537057 11581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:08.540161 11581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:08.540376 11581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:08.549111 11581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41551
I20260927 16:35:08.549130 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41551 every 8 connection(s)
I20260927 16:35:08.550077 11581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:35:08.552886 11649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:08.557289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 11581
I20260927 16:35:08.557425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:35:08.557744 392 external_mini_cluster.cc:1428] Setting key 906d088e372a68702245c5d3554a1322
I20260927 16:35:08.559674 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: Bootstrap starting.
I20260927 16:35:08.562355 11649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:08.563504 11649 log.cc:824] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:08.565804 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: No bootstrap required, opened a new log
Sep 27 16:35:08 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:08.570430 11649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 16:35:08.570778 11649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:08.570852 11649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292daa1fa46544fcbd7c3f94c344737c, State: Initialized, Role: FOLLOWER
I20260927 16:35:08.571384 11649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 16:35:08.571594 11649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:35:08.571667 11649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:35:08.571762 11649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:08.572942 11649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 16:35:08.572993 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:08.559240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.0.98.62:41551)
0927 16:35:08.559628 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:08.559648 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:08.559692 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:35:08.560536 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:08.560558 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:08.561022 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:08.561391 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:08.562335 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:08.563929 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:08.564908 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:08.565409 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:08.565588 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:08.568106 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:08.568111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:08.568125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:08.568152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:08.569819 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:08.570327 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:08.570335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:08.570346 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:08.570452 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:08.570819 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:08.570822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:08.570823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:08.571364 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:08.571575 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:08.571967 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 16:35:08.572282 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260927 16:35:08.573377 11649 leader_election.cc:302] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 292daa1fa46544fcbd7c3f94c344737c; no voters:
I20260927 16:35:08.573750 11649 leader_election.cc:288] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:35:08.573904 11654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:35:08.574194 11654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [term 1 LEADER]: Becoming Leader. State: Replica: 292daa1fa46544fcbd7c3f94c344737c, State: Running, Role: LEADER
I20260927 16:35:08.574543 11654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 16:35:08.575158 11649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:35:08.575834 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "292daa1fa46544fcbd7c3f94c344737c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260927 16:35:08.575963 11655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 292daa1fa46544fcbd7c3f94c344737c. Latest consensus state: current_term: 1 leader_uuid: "292daa1fa46544fcbd7c3f94c344737c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292daa1fa46544fcbd7c3f94c344737c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260927 16:35:08.576021 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:08.576098 11655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:08.577657 11649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:35:08.578418 11663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:35:08.584972 11663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:35:08.593004 11663 catalog_manager.cc:1347] Generated new cluster ID: b133b205f58a4dc2992f80a9fc394660
I20260927 16:35:08.593181 11663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:35:08.608526 11663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:35:08.611028 11663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:35:08.631146 11663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: Generated new TSK 0
I20260927 16:35:08.632216 11663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:35:09.289923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:09.447173 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:09.447609 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:09.447712 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:09.453853 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:09.454003 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:09.454116 11694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:35:09.462353 11694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:09.464020 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:09.465359 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:09.476087 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:09.476112 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:09.476107 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:09.476498 11694 server_base.cc:1034] running on GCE node
I20260927 16:35:09.476881 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:09.477654 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:09.485004 11694 hybrid_clock.cc:648] HybridClock initialized: now 1790526909484949 us; error 60 us; skew 500 ppm
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:09.488166 11694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:35:09.489307 11694 webserver.cc:460] Webserver started at http://127.0.98.1:46623/ using document root <none> and password file <none>
I20260927 16:35:09.490026 11694 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:09.490103 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:09.490362 11694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:09.520234 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "86806380f0eb409bad9cac66d61780fc"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "e9e66290a6a6f120a950a64c244a2451"
server_key_iv: "3cc2396b33c7f50f965499971f03ed4c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.521102 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "86806380f0eb409bad9cac66d61780fc"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "e9e66290a6a6f120a950a64c244a2451"
server_key_iv: "3cc2396b33c7f50f965499971f03ed4c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.526464 11694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:35:09.550585 11713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.559489 11694 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.000s
I20260927 16:35:09.559675 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "86806380f0eb409bad9cac66d61780fc"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "e9e66290a6a6f120a950a64c244a2451"
server_key_iv: "3cc2396b33c7f50f965499971f03ed4c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.559815 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:09.580235 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:09.584131 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:09.584399 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:09.585134 11694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:09.586158 11694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:09.586292 11694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.586411 11694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:09.586486 11694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.598129 11694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37991
I20260927 16:35:09.598340 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37991 every 8 connection(s)
I20260927 16:35:09.599221 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:35:09.603966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 11694
I20260927 16:35:09.604106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:35:09.604398 392 external_mini_cluster.cc:1428] Setting key c3cc48ba8c8cdb0a837a8c660e600e7b
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:35:09.615811 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:09.601117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36849 (local address 127.0.98.62:41551)
0927 16:35:09.601440 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:09.601449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:09.602308 (+ 859us) server_negotiation.cc:407] Connection header received
0927 16:35:09.603314 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:09.603318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:09.603378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:09.603498 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:09.605393 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.606143 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.608006 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.608338 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.608455 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:09.611443 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:09.611446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:09.611448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:09.611471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:09.613095 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:09.613773 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:09.613776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:09.613778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:09.613879 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:09.614368 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:09.614371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:09.614373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:09.614623 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:09.614697 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:09.615491 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 16:35:09.615630 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260927 16:35:09.616799 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:09.601652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.1:36849)
0927 16:35:09.602126 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:09.602202 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:09.603095 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:09.603722 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:09.603733 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:09.604106 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:09.605228 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:09.605252 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.606335 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.606338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:09.607880 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:09.607890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.608540 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.608545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:09.608779 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:09.609492 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:09.609517 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:09.611258 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:09.613222 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:09.613229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:09.613247 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:09.613643 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:09.614001 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:09.614004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:09.614006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:09.614254 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:09.614781 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:09.614792 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:09.615174 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 16:35:09.615434 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 16:35:09.615788 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260927 16:35:09.618223 11827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:35:09.618498 11827 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:09.619129 11827 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 16:35:09.621511 11613 ts_manager.cc:194] Registered new tserver with Master: 86806380f0eb409bad9cac66d61780fc (127.0.98.1:37991)
I20260927 16:35:09.622726 11613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:35:09.638057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:09.803045 11833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:09.803397 11833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:09.803465 11833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:09.810052 11833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:09.810138 11833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:09.810247 11833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:35:09.818359 11833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:09.819777 11833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:09.821154 11833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:09.833894 11840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:09.834110 11842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:09.833904 11839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:09.834983 11833 server_base.cc:1034] running on GCE node
I20260927 16:35:09.835455 11833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:09.836298 11833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:09.837502 11833 hybrid_clock.cc:648] HybridClock initialized: now 1790526909837475 us; error 41 us; skew 500 ppm
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:09.841596 11833 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:35:09.842744 11833 webserver.cc:460] Webserver started at http://127.0.98.2:43783/ using document root <none> and password file <none>
I20260927 16:35:09.843478 11833 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:09.843643 11833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:09.844161 11833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:09.865975 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "ccd3058ed8124927af55bb1b87ef5dc3"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "cfa599fdc1d6608203db6626cf624486"
server_key_iv: "5dde63cdce19df67060f9dee3964d470"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.866711 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ccd3058ed8124927af55bb1b87ef5dc3"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "cfa599fdc1d6608203db6626cf624486"
server_key_iv: "5dde63cdce19df67060f9dee3964d470"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.870957 11833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:35:09.889154 11849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.890923 11833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:35:09.891137 11833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "ccd3058ed8124927af55bb1b87ef5dc3"
format_stamp: "Formatted at 2026-09-27 16:35:09 on dist-test-slave-t27c"
server_key: "cfa599fdc1d6608203db6626cf624486"
server_key_iv: "5dde63cdce19df67060f9dee3964d470"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:09.891388 11833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:09.927213 11833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:09.931435 11833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:09.931720 11833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:09.932571 11833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:09.933764 11833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:09.933895 11833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.934021 11833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:09.934098 11833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:09.948647 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36229 every 8 connection(s)
I20260927 16:35:09.948833 11833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36229
I20260927 16:35:09.950031 11833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:35:09.958048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 11833
I20260927 16:35:09.958218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:35:09.958521 392 external_mini_cluster.cc:1428] Setting key e58fb3d7ebfc4aa829f14c0ce5486eac
Sep 27 16:35:09 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:09.968132 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:09.954225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42489 (local address 127.0.98.62:41551)
0927 16:35:09.954354 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:09.954359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:09.954794 (+ 435us) server_negotiation.cc:407] Connection header received
0927 16:35:09.955989 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:09.955993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:09.956058 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:09.956110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:09.957668 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.958685 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.960413 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.960758 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.960835 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:09.963851 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:09.963855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:09.963858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:09.963884 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:09.965677 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:09.966382 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:09.966385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:09.966387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:09.966470 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:09.966851 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:09.966854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:09.966856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:09.967079 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:09.967160 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:09.967828 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 16:35:09.967956 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:35:09.969901 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:09.953094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.2:42489)
0927 16:35:09.954636 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:09.954690 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:09.955749 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:09.956493 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:09.956501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:09.956839 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:09.957491 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:09.957510 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.958885 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.958889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:09.960270 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:09.960284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:09.961045 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:09.961049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:09.961246 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:09.961927 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:09.961947 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:09.963667 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:09.965871 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:09.965877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:09.965894 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:09.966269 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:09.966573 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:09.966576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:09.966578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:09.966743 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:09.967224 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:09.967231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:09.967555 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 16:35:09.968552 (+ 997us) client_negotiation.cc:240] Negotiation successful
0927 16:35:09.968875 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":143,"threads_started":1}
I20260927 16:35:09.971380 11963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260927 16:35:09.971608 11963 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:09.972139 11963 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 16:35:09.973222 11613 ts_manager.cc:194] Registered new tserver with Master: ccd3058ed8124927af55bb1b87ef5dc3 (127.0.98.2:36229)
I20260927 16:35:09.973944 11613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42489
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:35:10.013659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:35:10 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 16:35:10.237551 11968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:10.237999 11968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:10.238097 11968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:10.244078 11968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:10.244251 11968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:10.244387 11968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:35:10.254196 11968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:10.255939 11968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:10.257345 11968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:10.269953 11973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:10.269981 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:10.270700 11976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:10.271409 11968 server_base.cc:1034] running on GCE node
I20260927 16:35:10.271915 11968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:10.272750 11968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:10.273954 11968 hybrid_clock.cc:648] HybridClock initialized: now 1790526910273929 us; error 38 us; skew 500 ppm
Sep 27 16:35:10 dist-test-slave-t27c krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:10.277762 11968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:35:10.278915 11968 webserver.cc:460] Webserver started at http://127.0.98.3:40005/ using document root <none> and password file <none>
I20260927 16:35:10.279659 11968 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:10.279757 11968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:10.280061 11968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:10 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:10.296007 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "c881a419bc7e405c985cfe23ca776323"
format_stamp: "Formatted at 2026-09-27 16:35:10 on dist-test-slave-t27c"
server_key: "8ed023b25bd15e6099d7ba87ff54a07a"
server_key_iv: "8df2a9b342de0a1b5926b26c1ccbf67a"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:10.297107 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c881a419bc7e405c985cfe23ca776323"
format_stamp: "Formatted at 2026-09-27 16:35:10 on dist-test-slave-t27c"
server_key: "8ed023b25bd15e6099d7ba87ff54a07a"
server_key_iv: "8df2a9b342de0a1b5926b26c1ccbf67a"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:10.314906 11968 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.006s sys 0.000s
I20260927 16:35:10.332618 11983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:10.334308 11968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:35:10.334481 11968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "c881a419bc7e405c985cfe23ca776323"
format_stamp: "Formatted at 2026-09-27 16:35:10 on dist-test-slave-t27c"
server_key: "8ed023b25bd15e6099d7ba87ff54a07a"
server_key_iv: "8df2a9b342de0a1b5926b26c1ccbf67a"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:10.334614 11968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:10.362411 11968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:10.366230 11968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:10.366499 11968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:10.367194 11968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:10.368247 11968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:10.368350 11968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:10.368444 11968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:10.368512 11968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:10.382050 11968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46723
I20260927 16:35:10.383347 11968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:35:10.384616 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46723 every 8 connection(s)
I20260927 16:35:10.392063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 11968
I20260927 16:35:10.392225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:35:10.392577 392 external_mini_cluster.cc:1428] Setting key a4fa099871fb744ab3fd90add57e8a50
Sep 27 16:35:10 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:10.402648 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.388234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57105 (local address 127.0.98.62:41551)
0927 16:35:10.388385 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.388391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.389310 (+ 919us) server_negotiation.cc:407] Connection header received
0927 16:35:10.390566 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.390570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.390626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.390679 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:10.392350 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.393122 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.394865 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.395186 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.395280 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.398349 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:10.398353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:10.398356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:10.398383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:10.400152 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:10.400798 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:10.400802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:10.400804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:10.400889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:10.401293 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:10.401296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:10.401298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:10.401509 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:10.401591 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.402348 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.402484 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:35:10.404672 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.386918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.3:57105)
0927 16:35:10.389126 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.389179 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.390306 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.391094 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.391103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.391465 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:10.392106 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.392124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.393355 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.393358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.394727 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.394738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.395492 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.395496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.395689 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.396396 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:10.396417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:10.398130 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:10.400323 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:10.400329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:10.400346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:10.400681 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:10.400996 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:10.400999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:10.401001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:10.401172 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:10.401655 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:10.401663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:10.402034 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.403198 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.403572 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990,"thread_start_us":125,"threads_started":1}
I20260927 16:35:10.406288 12097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260927 16:35:10.406543 12097 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:10.407105 12097 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 16:35:10.408373 11612 ts_manager.cc:194] Registered new tserver with Master: c881a419bc7e405c985cfe23ca776323 (127.0.98.3:46723)
I20260927 16:35:10.409029 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57105
I20260927 16:35:10.418637 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:35:10.432128 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.421543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.98.62:41551)
0927 16:35:10.421666 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.421671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.422303 (+ 632us) server_negotiation.cc:407] Connection header received
0927 16:35:10.422518 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.422521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.422572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.422626 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:10.423623 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.424483 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.425722 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.426084 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.426162 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.428123 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:10.428128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:10.428130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:10.428160 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:10.429989 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:10.430542 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:10.430545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:10.430547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:10.430632 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:10.431025 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:10.431029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:10.431031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:10.431278 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:10.431397 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.431771 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.431951 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:35:10.436395 11612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:35:10.441056 11612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:35:10.476548 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.456653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:51254)
0927 16:35:10.460696 (+ 4043us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.460708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.460782 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.462564 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.462567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.462582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:10.462669 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.462674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.466294 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.466298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.468615 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.468627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.475946 (+ 7319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.475950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.476089 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.476135 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.476253 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.476387 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3985,"thread_start_us":94,"threads_started":1}
I20260927 16:35:10.479323 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.455414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46723 (local address 127.0.0.1:48026)
0927 16:35:10.458455 (+ 3041us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.458467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.458539 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.462098 (+ 3559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.462101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.462122 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:10.462240 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.462246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.470152 (+ 7906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.470156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.472217 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.472227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.478907 (+ 6680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.478911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.479013 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.479043 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.479106 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.479180 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2982,"thread_start_us":94,"threads_started":1}
I20260927 16:35:10.480684 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.456857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.0.98.1:37991)
0927 16:35:10.458780 (+ 1923us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.458788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.460887 (+ 2099us) server_negotiation.cc:407] Connection header received
0927 16:35:10.460966 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.460971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.461160 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.462421 (+ 1261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:10.463503 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.464714 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.472367 (+ 7653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.473291 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.473401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.473571 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.480411 (+ 6840us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.480542 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1804,"thread_start_us":94,"threads_started":1}
I20260927 16:35:10.481359 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.453306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36229 (local address 127.0.0.1:38134)
0927 16:35:10.457991 (+ 4685us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.458046 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.458197 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.460328 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.460335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.460359 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:10.460497 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.460503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.473660 (+ 13157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.473664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.474928 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.474939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.480994 (+ 6055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.480999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.481095 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.481119 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.481167 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.481231 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4478,"thread_start_us":90,"threads_started":1}
I20260927 16:35:10.482344 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.453452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.98.2:36229)
0927 16:35:10.458983 (+ 5531us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.459000 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.459027 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:35:10.459105 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.459110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.459348 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.459501 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:10.464968 (+ 5467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.466149 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.475114 (+ 8965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.479594 (+ 4480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.481802 (+ 2208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.482036 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.482091 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.482211 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5423,"thread_start_us":86,"threads_started":1}
I20260927 16:35:10.484586 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.455845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.0.98.3:46723)
0927 16:35:10.461489 (+ 5644us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.461501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.461522 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:35:10.461593 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.461598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.461795 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.461953 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:10.468787 (+ 6834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.470006 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.477676 (+ 7670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.478583 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.478691 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.478846 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.483523 (+ 4677us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.484434 (+ 911us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5536,"thread_start_us":166,"threads_started":1}
I20260927 16:35:10.485858 11760 tablet_service.cc:1467] Processing CreateTablet for tablet adbef085c47f4f9abb9653dcf38a6306 (DEFAULT_TABLE table=test-table [id=1c7d1b3b6e7843488b79b3a03fa25407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:10.487726 12031 tablet_service.cc:1467] Processing CreateTablet for tablet adbef085c47f4f9abb9653dcf38a6306 (DEFAULT_TABLE table=test-table [id=1c7d1b3b6e7843488b79b3a03fa25407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:10.488911 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbef085c47f4f9abb9653dcf38a6306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:10.489074 11881 tablet_service.cc:1467] Processing CreateTablet for tablet adbef085c47f4f9abb9653dcf38a6306 (DEFAULT_TABLE table=test-table [id=1c7d1b3b6e7843488b79b3a03fa25407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:10.490036 11881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbef085c47f4f9abb9653dcf38a6306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:10.496914 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbef085c47f4f9abb9653dcf38a6306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:10.499049 12116 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Bootstrap starting.
I20260927 16:35:10.502197 12116 tablet_bootstrap.cc:654] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:10.504177 12116 log.cc:824] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:10.506376 12116 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: No bootstrap required, opened a new log
I20260927 16:35:10.506577 12116 ts_tablet_manager.cc:1397] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 16:35:10.510618 12116 raft_consensus.cc:348] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.511014 12116 raft_consensus.cc:374] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:10.511075 12116 raft_consensus.cc:729] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c881a419bc7e405c985cfe23ca776323, State: Initialized, Role: FOLLOWER
I20260927 16:35:10.511860 12116 consensus_queue.cc:260] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.513157 12116 ts_tablet_manager.cc:1428] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 16:35:10.513572 12097 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
I20260927 16:35:10.513599 12119 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Bootstrap starting.
I20260927 16:35:10.515990 12119 tablet_bootstrap.cc:654] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:10.516806 12119 log.cc:824] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:10.517514 12117 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Bootstrap starting.
I20260927 16:35:10.522193 12119 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: No bootstrap required, opened a new log
I20260927 16:35:10.522485 12119 ts_tablet_manager.cc:1397] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 16:35:10.527024 12119 raft_consensus.cc:348] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.527419 12119 raft_consensus.cc:374] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:10.527527 12119 raft_consensus.cc:729] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86806380f0eb409bad9cac66d61780fc, State: Initialized, Role: FOLLOWER
I20260927 16:35:10.528128 12119 consensus_queue.cc:260] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.529307 12119 ts_tablet_manager.cc:1428] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 16:35:10.529732 11827 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
I20260927 16:35:10.531724 12117 tablet_bootstrap.cc:654] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:10.532613 12117 log.cc:824] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:10.534868 12117 tablet_bootstrap.cc:492] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: No bootstrap required, opened a new log
I20260927 16:35:10.535063 12117 ts_tablet_manager.cc:1397] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260927 16:35:10.540161 12117 raft_consensus.cc:348] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.540547 12117 raft_consensus.cc:374] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:10.540649 12117 raft_consensus.cc:729] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd3058ed8124927af55bb1b87ef5dc3, State: Initialized, Role: FOLLOWER
I20260927 16:35:10.541294 12117 consensus_queue.cc:260] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.542472 12117 ts_tablet_manager.cc:1428] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 16:35:10.542868 11963 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
Sep 27 16:35:10 dist-test-slave-t27c krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 16:35:10.600804 11828 tablet.cc:2369] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:35:10.639128 12098 tablet.cc:2369] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:35:10.675904 12124 raft_consensus.cc:482] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:35:10.676168 12124 raft_consensus.cc:504] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.677697 12124 leader_election.cc:288] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c881a419bc7e405c985cfe23ca776323 (127.0.98.3:46723), 86806380f0eb409bad9cac66d61780fc (127.0.98.1:37991)
I20260927 16:35:10.685755 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.679292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38685 (local address 127.0.98.3:46723)
0927 16:35:10.679551 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.679555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.681140 (+ 1585us) server_negotiation.cc:407] Connection header received
0927 16:35:10.681268 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.681272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.681331 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.681424 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:10.681878 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.682710 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.684412 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.685034 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.685117 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.685188 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.685532 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.685611 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260927 16:35:10.686197 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.678218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46723 (local address 127.0.98.2:38685)
0927 16:35:10.680992 (+ 2774us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.681009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.681160 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.681545 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.681548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.681567 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:10.681703 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.681709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.682893 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.682896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.684260 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.684272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.685284 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.685288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.685387 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.685410 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.685937 (+ 527us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.686027 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"thread_start_us":168,"threads_started":1}
I20260927 16:35:10.687409 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbef085c47f4f9abb9653dcf38a6306" candidate_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c881a419bc7e405c985cfe23ca776323" is_pre_election: true
I20260927 16:35:10.687837 12051 raft_consensus.cc:2454] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccd3058ed8124927af55bb1b87ef5dc3 in term 0.
I20260927 16:35:10.688462 11851 leader_election.cc:302] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c881a419bc7e405c985cfe23ca776323, ccd3058ed8124927af55bb1b87ef5dc3; no voters:
I20260927 16:35:10.688817 12124 raft_consensus.cc:2781] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:35:10.688894 12124 raft_consensus.cc:482] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:35:10.688935 12124 raft_consensus.cc:3037] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:10.690598 12124 raft_consensus.cc:504] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.691025 12124 leader_election.cc:288] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [CANDIDATE]: Term 1 election: Requested vote from peers c881a419bc7e405c985cfe23ca776323 (127.0.98.3:46723), 86806380f0eb409bad9cac66d61780fc (127.0.98.1:37991)
I20260927 16:35:10.691110 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.679418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35447 (local address 127.0.98.1:37991)
0927 16:35:10.679640 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.679644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.686467 (+ 6823us) server_negotiation.cc:407] Connection header received
0927 16:35:10.686824 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.686827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.686878 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.686920 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:10.687368 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.688174 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.689750 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.690392 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.690475 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.690541 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.690898 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.690983 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 16:35:10.691516 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.678710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.98.2:35447)
0927 16:35:10.686353 (+ 7643us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.686369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.686552 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.687073 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:10.687076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:10.687094 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:10.687215 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.687221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.688320 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.688324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.689604 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:10.689616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.690637 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.690640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:10.690742 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.690765 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:35:10.691300 (+ 535us) client_negotiation.cc:240] Negotiation successful
0927 16:35:10.691378 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7568,"thread_start_us":85,"threads_started":1}
I20260927 16:35:10.691769 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbef085c47f4f9abb9653dcf38a6306" candidate_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c881a419bc7e405c985cfe23ca776323"
I20260927 16:35:10.691921 12051 raft_consensus.cc:3037] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:10.693334 12051 raft_consensus.cc:2454] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccd3058ed8124927af55bb1b87ef5dc3 in term 1.
I20260927 16:35:10.693853 11851 leader_election.cc:302] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c881a419bc7e405c985cfe23ca776323, ccd3058ed8124927af55bb1b87ef5dc3; no voters:
I20260927 16:35:10.694096 12124 raft_consensus.cc:2781] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:35:10.694088 11769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbef085c47f4f9abb9653dcf38a6306" candidate_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86806380f0eb409bad9cac66d61780fc" is_pre_election: true
I20260927 16:35:10.694357 12124 raft_consensus.cc:686] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 1 LEADER]: Becoming Leader. State: Replica: ccd3058ed8124927af55bb1b87ef5dc3, State: Running, Role: LEADER
I20260927 16:35:10.694515 11769 raft_consensus.cc:2454] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccd3058ed8124927af55bb1b87ef5dc3 in term 0.
I20260927 16:35:10.694730 12124 consensus_queue.cc:237] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260927 16:35:10.694839 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbef085c47f4f9abb9653dcf38a6306" candidate_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86806380f0eb409bad9cac66d61780fc"
I20260927 16:35:10.694939 11765 raft_consensus.cc:3037] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:10.696060 11765 raft_consensus.cc:2454] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccd3058ed8124927af55bb1b87ef5dc3 in term 1.
I20260927 16:35:10.698433 11612 catalog_manager.cc:5495] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccd3058ed8124927af55bb1b87ef5dc3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccd3058ed8124927af55bb1b87ef5dc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } health_report { overall_health: UNKNOWN } } }
W20260927 16:35:10.701531 11964 tablet.cc:2369] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:35:10.771273 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.765469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.2:36229)
0927 16:35:10.765626 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:10.765631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:10.765647 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:35:10.765747 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:10.765750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:10.765830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:10.765952 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:35:10.766393 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.767180 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.768492 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:10.768827 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:10.768933 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:10.769399 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:35:10.769974 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:35:10.770113 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:10.771042 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 16:35:10.771123 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 16:35:10.789248 11765 raft_consensus.cc:1261] T adbef085c47f4f9abb9653dcf38a6306 P 86806380f0eb409bad9cac66d61780fc [term 1 FOLLOWER]: Refusing update from remote peer ccd3058ed8124927af55bb1b87ef5dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:35:10.789404 12051 raft_consensus.cc:1261] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 1 FOLLOWER]: Refusing update from remote peer ccd3058ed8124927af55bb1b87ef5dc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:35:10.790266 12124 consensus_queue.cc:1035] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86806380f0eb409bad9cac66d61780fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:35:10.790762 12129 consensus_queue.cc:1035] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c881a419bc7e405c985cfe23ca776323" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:35:10.818001 12135 mvcc.cc:205] Tried to move back new op lower bound from 7333998226555305984 to 7333998226207920128. Current Snapshot: MvccSnapshot[applied={T|T < 7333998226555305984}]
I20260927 16:35:10.829977 11612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:35:10.830216 11612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:35:10.833277 11612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292daa1fa46544fcbd7c3f94c344737c: Sending DeleteTablet for 3 replicas of tablet adbef085c47f4f9abb9653dcf38a6306
I20260927 16:35:10.834236 12031 tablet_service.cc:1514] Processing DeleteTablet for tablet adbef085c47f4f9abb9653dcf38a6306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48026
I20260927 16:35:10.834261 11881 tablet_service.cc:1514] Processing DeleteTablet for tablet adbef085c47f4f9abb9653dcf38a6306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38134
I20260927 16:35:10.835223 11760 tablet_service.cc:1514] Processing DeleteTablet for tablet adbef085c47f4f9abb9653dcf38a6306 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51254
I20260927 16:35:10.836310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 11694
I20260927 16:35:10.842475 12142 tablet_replica.cc:330] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: stopping tablet replica
I20260927 16:35:10.845032 12142 raft_consensus.cc:2229] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:35:10.845716 12142 raft_consensus.cc:2258] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:35:10.850288 12142 ts_tablet_manager.cc:1905] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:35:10.859931 12142 ts_tablet_manager.cc:1918] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:35:10.860070 12142 log.cc:1196] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/adbef085c47f4f9abb9653dcf38a6306
I20260927 16:35:10.860400 12142 ts_tablet_manager.cc:1939] T adbef085c47f4f9abb9653dcf38a6306 P ccd3058ed8124927af55bb1b87ef5dc3: Deleting consensus metadata
I20260927 16:35:10.861832 12141 tablet_replica.cc:330] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323: stopping tablet replica
I20260927 16:35:10.861832 11597 catalog_manager.cc:4841] TS ccd3058ed8124927af55bb1b87ef5dc3 (127.0.98.2:36229): tablet adbef085c47f4f9abb9653dcf38a6306 (table test-table [id=1c7d1b3b6e7843488b79b3a03fa25407]) successfully deleted
I20260927 16:35:10.862296 12141 raft_consensus.cc:2229] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:35:10.862787 12141 raft_consensus.cc:2258] T adbef085c47f4f9abb9653dcf38a6306 P c881a419bc7e405c985cfe23ca776323 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:35:10.863389 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37991 (error 108)
I20260927 16:35:10.864894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 11833
I20260927 16:35:10.864971 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:10.864638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:51258)
0927 16:35:10.864792 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.864877 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:35:10.866392 11599 catalog_manager.cc:4584] TS 86806380f0eb409bad9cac66d61780fc (127.0.98.1:37991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adbef085c47f4f9abb9653dcf38a6306: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
I20260927 16:35:10.871768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 11968
W20260927 16:35:10.879654 12112 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:35:10.879019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46723 (local address 127.0.0.1:48040)
0927 16:35:10.879206 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:10.879222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:10.879320 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:10.879521 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:46723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 16:35:10.880419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 11581
I20260927 16:35:10.886773 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:35:11.221650 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:35:11.221797 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:35:11.566531 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:35:11.566 UTC [11167] LOG: received smart shutdown request
2026-09-27 16:35:11.570 UTC [11167] LOG: background worker "logical replication launcher" (PID 11176) exited with exit code 1
2026-09-27 16:35:11.571 UTC [11171] LOG: shutting down
2026-09-27 16:35:11.741 UTC [11167] LOG: database system is shut down
2026-09-27T16:35:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47183 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:35:11 dist-test-slave-t27c krb5kdc[12155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:35:11 dist-test-slave-t27c krb5kdc[12155](info): set up 2 sockets
Sep 27 16:35:11 dist-test-slave-t27c krb5kdc[12155](info): commencing operation
krb5kdc: starting...
W20260927 16:35:12.910360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:35:12 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:35:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 16:35:13.076387 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxe3jhj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/postgres -l logfile start
2026-09-27 16:35:15.477 UTC [12185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:35:15.477 UTC [12185] LOG: listening on IPv4 address "127.0.98.20", port 40919
2026-09-27 16:35:15.480 UTC [12185] LOG: listening on Unix socket "/tmp/.s.PGSQL.40919"
2026-09-27 16:35:15.506 UTC [12188] LOG: database system was shut down at 2026-09-27 16:35:14 UTC
2026-09-27 16:35:15.510 UTC [12185] LOG: database system is ready to accept connections
I20260927 16:35:16.478154 392 mini_postgres.cc:96] Postgres bound to 40919
2026-09-27 16:35:16.485 UTC [12196] FATAL: database "slave" does not exist
127.0.98.20:40919 - accepting connections
I20260927 16:35:16.485697 392 mini_ranger.cc:161] Starting Ranger...
I20260927 16:35:16.500108 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:35:16.951416 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:35:16.951490 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin
2026-09-27 16:35:17,236 [I] DB FLAVOR :POSTGRES
2026-09-27 16:35:17,237 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:35:17,237 [I] Checking connection..
2026-09-27 16:35:17,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:35:17,510 [I] Checking connection passed.
2026-09-27 16:35:17,510 [I] --------- Verifying version history table ---------
2026-09-27 16:35:17,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:35:17,740 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:35:17,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:35:17,966 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:35:17,966 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:35:17,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:35:18.198 UTC [12260] WARNING: there is no transaction in progress
2026-09-27 16:35:18,204 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:35:18,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:35:18,466 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:35:18,466 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:35:18,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:35:18,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:35:18,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t27c.c.gcp-upstream.internal','N') ;"
2026-09-27 16:35:19,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:35:19,435 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:35:19,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:35:19,663 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:35:19,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:35:19,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:35:20,160 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:35:20,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:35:21.100 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.115 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.131 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.146 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.162 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.181 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.239 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.246 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.260 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.269 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.279 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.293 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.302 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.309 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.315 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21.322 UTC [12368] WARNING: there is no transaction in progress
2026-09-27 16:35:21,933 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:35:21,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:35:22,153 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:35:22,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:35:22,439 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:35:22,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:35:22,669 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:35:22,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:35:22,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:35:23,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t27c.c.gcp-upstream.internal';"
2026-09-27 16:35:23,431 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:35:23,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:35:23,691 [I] DB_PATCHES have already been applied
I20260927 16:35:23.697434 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxe3jhj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:35:23.697507 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 16:35:23.700944 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41157
Sep 27, 2026 4:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:35:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:35:24.812563 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.327s user 0.001s sys 0.009s
Sep 27, 2026 4:35:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 4:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:35:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:35:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:35:46.519104 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:35:46 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526946, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:35:46.611023 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:35:46.620332 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:35:47.458595 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:35:47.458683 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms
I20260927 16:35:47.458705 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40919
2026-09-27 16:35:47,630 [I] DB FLAVOR :POSTGRES
2026-09-27 16:35:47,630 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:35:47,631 [I] Checking connection
2026-09-27 16:35:47,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:35:47,857 [I] connection success
2026-09-27 16:35:47,857 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:35:47,858 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:35:47,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:35:48,082 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:35:48,082 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:35:48,082 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:35:48,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:35:48,342 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:35:48.800220 392 mini_ranger_kms.cc:310] Created kms service
I20260927 16:35:48.999588 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:35:49.043286 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:35:49 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526946, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:49.427373 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:35:49.427524 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxe3jhj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxe3jhj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:35:49.427544 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 16:35:49.431643 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12548
I20260927 16:35:49.431731 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51249
Sep 27, 2026 4:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:35:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:35:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 16:35:51.694707 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.084s user 0.004s sys 0.151s
I20260927 16:35:51.694823 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:35:51.694885 392 mini_ranger_kms.cc:401] 127.0.98.20:51249/kms/v1/keys
Sep 27 16:35:51 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 16:35:52 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:35:52 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:35:54 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:35:54.806267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41713
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:54.916390 12599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:54.916659 12599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:54.916698 12599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:54.920588 12599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:35:54.920642 12599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:54.920661 12599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:54.920677 12599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:35:54.920692 12599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:35:54.925997 12599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51249/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41713
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:54.927060 12599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:54.928056 12599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:54.936838 12604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:54.936832 12605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:54.936827 12607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:54.936909 12599 server_base.cc:1034] running on GCE node
I20260927 16:35:54.937589 12599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:54.938766 12599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:54.939929 12599 hybrid_clock.cc:648] HybridClock initialized: now 1790526954939913 us; error 30 us; skew 500 ppm
Sep 27 16:35:54 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:54.947489 12599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:35:54.948410 12599 webserver.cc:460] Webserver started at http://127.0.98.62:40753/ using document root <none> and password file <none>
I20260927 16:35:54.948971 12599 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:54.949051 12599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:54.949278 12599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:54 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:55.002473 12599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "8f0fbe408dd249c084a49bc2d3a350fb"
format_stamp: "Formatted at 2026-09-27 16:35:54 on dist-test-slave-t27c"
server_key: "45cdeae6e0d5b2de6172f3e5a66fbfe6"
server_key_iv: "620ac0477eac339314a70c265aa4b80b"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:55.003126 12599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f0fbe408dd249c084a49bc2d3a350fb"
format_stamp: "Formatted at 2026-09-27 16:35:54 on dist-test-slave-t27c"
server_key: "45cdeae6e0d5b2de6172f3e5a66fbfe6"
server_key_iv: "620ac0477eac339314a70c265aa4b80b"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:55.007117 12599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:35:55.024283 12614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:55.025712 12599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:35:55.025863 12599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8f0fbe408dd249c084a49bc2d3a350fb"
format_stamp: "Formatted at 2026-09-27 16:35:54 on dist-test-slave-t27c"
server_key: "45cdeae6e0d5b2de6172f3e5a66fbfe6"
server_key_iv: "620ac0477eac339314a70c265aa4b80b"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:55.026001 12599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:55.060541 12599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:55.063153 12599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:55.063354 12599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:55.070919 12599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41713
I20260927 16:35:55.070930 12666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41713 every 8 connection(s)
I20260927 16:35:55.071873 12599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:35:55.073909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 12599
I20260927 16:35:55.074043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:35:55.074385 392 external_mini_cluster.cc:1428] Setting key 6fe7c0cccaff98f44b58d9cf8c4595cc
I20260927 16:35:55.074945 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:55.081609 12667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Bootstrap starting.
Sep 27 16:35:55 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:55.084324 12667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:55.085110 12667 log.cc:824] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:55.087167 12667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: No bootstrap required, opened a new log
I20260927 16:35:55.089484 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:55.075806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.0.98.62:41713)
0927 16:35:55.076189 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:55.076215 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:55.076249 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:35:55.076905 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:55.076922 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:55.077242 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:55.077517 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:55.078536 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:55.079534 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:55.080716 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:55.081179 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:55.081291 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:55.084250 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:55.084255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:55.084269 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:55.084302 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:55.086716 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:55.087198 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:55.087210 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:55.087215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:55.087366 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:55.087650 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:55.087653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:55.087655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:55.088040 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:55.088263 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:55.088533 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 16:35:55.088804 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260927 16:35:55.090372 12667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:55.090632 12667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:55.090694 12667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0fbe408dd249c084a49bc2d3a350fb, State: Initialized, Role: FOLLOWER
I20260927 16:35:55.091158 12667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:55.091305 12667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:35:55.091368 12667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:35:55.091459 12667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:55.092471 12667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:55.092764 12667 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0fbe408dd249c084a49bc2d3a350fb; no voters:
I20260927 16:35:55.093030 12667 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:35:55.093144 12672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:35:55.093410 12672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 LEADER]: Becoming Leader. State: Replica: 8f0fbe408dd249c084a49bc2d3a350fb, State: Running, Role: LEADER
I20260927 16:35:55.093716 12672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:55.094245 12667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:35:55.095556 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0fbe408dd249c084a49bc2d3a350fb. Latest consensus state: current_term: 1 leader_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260927 16:35:55.095772 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260927 16:35:55.095865 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:55.095757 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:55.096371 12685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:35:55.096452 12667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:35:55.101547 12685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:35:55.107059 12685 catalog_manager.cc:1347] Generated new cluster ID: c5986065109644598e72f0eceb47bc13
I20260927 16:35:55.107121 12685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:35:55.148298 12685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:35:55.149355 12685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:35:55.156299 12685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Generated new TSK 0
I20260927 16:35:55.157298 12685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:35:55.784986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:55.926007 12712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:55.926378 12712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:55.926442 12712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:55.932556 12712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:55.932686 12712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:55.932808 12712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:35:55.940912 12712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:55.942476 12712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:55.943854 12712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:55.956012 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:55.956245 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:55.956524 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:55.956686 12712 server_base.cc:1034] running on GCE node
I20260927 16:35:55.957267 12712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:55.958073 12712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:55.959280 12712 hybrid_clock.cc:648] HybridClock initialized: now 1790526955959250 us; error 34 us; skew 500 ppm
Sep 27 16:35:55 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:55.962823 12712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:35:55.963702 12712 webserver.cc:460] Webserver started at http://127.0.98.1:42955/ using document root <none> and password file <none>
I20260927 16:35:55.964272 12712 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:55.964363 12712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:55.964613 12712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:55 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:55.980217 12712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "9137525e351f4ea7857b20e39ad9f741"
format_stamp: "Formatted at 2026-09-27 16:35:55 on dist-test-slave-t27c"
server_key: "1899e879d3bb78e5a5dd8cf7f5b12f9c"
server_key_iv: "bb290f1bd56aa50adbd24725b3b9c6fd"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:55.981071 12712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9137525e351f4ea7857b20e39ad9f741"
format_stamp: "Formatted at 2026-09-27 16:35:55 on dist-test-slave-t27c"
server_key: "1899e879d3bb78e5a5dd8cf7f5b12f9c"
server_key_iv: "bb290f1bd56aa50adbd24725b3b9c6fd"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:55.986641 12712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:35:56.014652 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.016894 12712 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260927 16:35:56.017068 12712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "9137525e351f4ea7857b20e39ad9f741"
format_stamp: "Formatted at 2026-09-27 16:35:55 on dist-test-slave-t27c"
server_key: "1899e879d3bb78e5a5dd8cf7f5b12f9c"
server_key_iv: "bb290f1bd56aa50adbd24725b3b9c6fd"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.017241 12712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:56.041458 12712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:56.045343 12712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:56.045593 12712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:56.046356 12712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:56.047397 12712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:56.047461 12712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.047511 12712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:56.047582 12712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 16:35:56.059156 12712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37331
I20260927 16:35:56.059228 12844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37331 every 8 connection(s)
I20260927 16:35:56.060372 12712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:35:56.065965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 12712
I20260927 16:35:56.066097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:35:56.066411 392 external_mini_cluster.cc:1428] Setting key 32b3c253f99152cf8ff7a6dddf9b05b6
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:56.077673 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.062616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52253 (local address 127.0.98.62:41713)
0927 16:35:56.062952 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:56.062962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:56.063867 (+ 905us) server_negotiation.cc:407] Connection header received
0927 16:35:56.065245 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:56.065249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:56.065307 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:56.065412 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:56.067122 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.068023 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.069773 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.070132 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.070285 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.073453 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:56.073457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:56.073460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:56.073485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:56.075387 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.075952 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.075956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.075958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.076037 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.076422 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.076425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.076427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.076623 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:56.076694 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:56.077389 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 16:35:56.077512 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260927 16:35:56.079586 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.063069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.1:52253)
0927 16:35:56.063689 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:56.063750 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:56.064966 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:56.065653 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:56.065662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:56.066053 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:56.066926 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.066949 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.068193 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.068197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.069633 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.069644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.070498 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.070502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.070811 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.071524 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:56.071548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:56.073264 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:56.075515 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.075520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.075536 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.075846 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.076139 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.076142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.076144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.076315 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.076779 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:56.076785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:56.077093 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 16:35:56.078122 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0927 16:35:56.078471 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":184,"threads_started":1}
I20260927 16:35:56.081106 12845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:56.081338 12845 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:56.081846 12845 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:56.083604 12631 ts_manager.cc:194] Registered new tserver with Master: 9137525e351f4ea7857b20e39ad9f741 (127.0.98.1:37331)
I20260927 16:35:56.085243 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52253
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:35:56.109016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:56.247413 12851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:56.247704 12851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:56.247840 12851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:56.252483 12851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:56.252579 12851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:56.252677 12851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:35:56.259052 12851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:56.260174 12851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:56.261500 12851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:56.272271 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:56.272264 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:56.272744 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:56.273440 12851 server_base.cc:1034] running on GCE node
I20260927 16:35:56.273875 12851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:56.274674 12851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:56.275830 12851 hybrid_clock.cc:648] HybridClock initialized: now 1790526956275764 us; error 82 us; skew 500 ppm
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:56.279309 12851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:35:56.280270 12851 webserver.cc:460] Webserver started at http://127.0.98.2:43843/ using document root <none> and password file <none>
I20260927 16:35:56.280905 12851 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:56.281035 12851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:56.281289 12851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:56.298448 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "5be0f07bbf874e56800b8b8b62050997"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "6a46bf6e8f2f78b55805df98938fb663"
server_key_iv: "6e68d6a3c51e620c4036c524e4a6e07c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.299239 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5be0f07bbf874e56800b8b8b62050997"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "6a46bf6e8f2f78b55805df98938fb663"
server_key_iv: "6e68d6a3c51e620c4036c524e4a6e07c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.304987 12851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:35:56.326481 12867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.327986 12851 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 16:35:56.328145 12851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "5be0f07bbf874e56800b8b8b62050997"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "6a46bf6e8f2f78b55805df98938fb663"
server_key_iv: "6e68d6a3c51e620c4036c524e4a6e07c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.328296 12851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:56.358968 12851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:56.362511 12851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:56.362716 12851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:56.363463 12851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:56.364595 12851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:56.364715 12851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.364820 12851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:56.364886 12851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.375351 12851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33637
I20260927 16:35:56.375480 12980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33637 every 8 connection(s)
I20260927 16:35:56.376528 12851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:35:56.384037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 12851
I20260927 16:35:56.384186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:35:56.384506 392 external_mini_cluster.cc:1428] Setting key 406c9544a505529f722ff5b2b9a59c49
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:56.392156 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.378510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41485 (local address 127.0.98.62:41713)
0927 16:35:56.378666 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:56.378671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:56.379608 (+ 937us) server_negotiation.cc:407] Connection header received
0927 16:35:56.380795 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:56.380798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:56.380843 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:56.380922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:56.382336 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.382862 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.384504 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.384830 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.384939 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.387833 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:56.387839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:56.387843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:56.387877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:56.389650 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.390294 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.390298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.390300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.390386 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.390799 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.390802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.390804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.391022 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:56.391105 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:56.391872 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 16:35:56.391999 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:35:56.394086 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.378898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.2:41485)
0927 16:35:56.379402 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:56.379471 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:56.380538 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:56.381106 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:56.381115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:56.381482 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:56.382153 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.382201 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.382997 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.383000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.384358 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.384372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.385008 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.385012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.385211 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.385885 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:56.385904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:56.387643 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:56.389783 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.389790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.389806 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.390137 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.390513 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.390516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.390518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.390690 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.391174 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:56.391181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:56.391584 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 16:35:56.392484 (+ 900us) client_negotiation.cc:240] Negotiation successful
0927 16:35:56.392842 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260927 16:35:56.395488 12981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:56.395738 12981 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:56.396394 12981 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:35:56.397570 12631 ts_manager.cc:194] Registered new tserver with Master: 5be0f07bbf874e56800b8b8b62050997 (127.0.98.2:33637)
I20260927 16:35:56.398442 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:35:56.420845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 16:35:56.580075 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:56.580408 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:56.580470 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:56.586401 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:56.586539 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:56.586666 12986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:35:56.594228 12986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:56.595764 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:56.597123 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:56.609746 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:56.610656 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:56.610924 12986 server_base.cc:1034] running on GCE node
W20260927 16:35:56.611227 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:56.611729 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:56.612517 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:56.613670 12986 hybrid_clock.cc:648] HybridClock initialized: now 1790526956613596 us; error 88 us; skew 500 ppm
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:56.617146 12986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:35:56.618225 12986 webserver.cc:460] Webserver started at http://127.0.98.3:38365/ using document root <none> and password file <none>
I20260927 16:35:56.618898 12986 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:56.618974 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:35:56.619189 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:56.635403 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "416bfd42868c4863bf6a8d2e2c6c6a06"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "9aa5b4c250b65080557244550f249ba4"
server_key_iv: "d909bbda3792c78285a42c97824a8ac5"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.636224 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "416bfd42868c4863bf6a8d2e2c6c6a06"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "9aa5b4c250b65080557244550f249ba4"
server_key_iv: "d909bbda3792c78285a42c97824a8ac5"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.655972 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.003s sys 0.000s
I20260927 16:35:56.675556 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.677404 12986 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 16:35:56.677565 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "416bfd42868c4863bf6a8d2e2c6c6a06"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "9aa5b4c250b65080557244550f249ba4"
server_key_iv: "d909bbda3792c78285a42c97824a8ac5"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.677712 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:56.696063 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:56.699594 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:56.699837 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:56.700525 12986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:56.701543 12986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:56.701655 12986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.701753 12986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:56.701810 12986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.715298 12986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38947
I20260927 16:35:56.716610 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:35:56.721808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 12986
I20260927 16:35:56.721944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:35:56.722265 392 external_mini_cluster.cc:1428] Setting key b08f9ee87a9c7aaa7f586e7f250eb18e
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:56.717886 13114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38947 every 8 connection(s)
I20260927 16:35:56.740159 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.721531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42361 (local address 127.0.98.62:41713)
0927 16:35:56.721689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:56.721694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:56.722262 (+ 568us) server_negotiation.cc:407] Connection header received
0927 16:35:56.725592 (+ 3330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:56.725597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:56.725661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:56.725723 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:56.727609 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.728510 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.730708 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.731015 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.731085 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.734237 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:56.734242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:56.734245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:56.734274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:56.737619 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.738245 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.738249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.738251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.738333 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:56.738865 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:56.738869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:56.738871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:56.739089 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:56.739174 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:56.739871 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 16:35:56.739994 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 16:35:56.742008 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:56.720324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.3:42361)
0927 16:35:56.722051 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:56.722110 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:56.725285 (+ 3175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:56.726227 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:56.726240 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:56.726650 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:56.727409 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.727434 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.728689 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.728694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.730571 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:56.730585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:56.731292 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:56.731297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:56.731507 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:56.732220 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:56.732242 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:56.734007 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:56.737752 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.737758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.737776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.738105 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.738571 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:56.738574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:56.738575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:56.738746 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:56.739240 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:56.739246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:56.739593 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 16:35:56.740592 (+ 999us) client_negotiation.cc:240] Negotiation successful
0927 16:35:56.740933 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"thread_start_us":126,"threads_started":1}
I20260927 16:35:56.743516 13115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:56.743747 13115 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:56.744300 13115 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:56.745487 12631 ts_manager.cc:194] Registered new tserver with Master: 416bfd42868c4863bf6a8d2e2c6c6a06 (127.0.98.3:38947)
I20260927 16:35:56.746273 12631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42361
I20260927 16:35:56.750232 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:35:56.751102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 12712
I20260927 16:35:56.759902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 12851
I20260927 16:35:56.766615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 12986
I20260927 16:35:56.774627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 12599
I20260927 16:35:56.780134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--webserver_interface=127.0.98.62
--webserver_port=40753
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41713
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:35:56.891085 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:56.891340 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:56.891377 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:56.895489 13118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:35:56.895546 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:56.895565 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:56.895581 13118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:35:56.895596 13118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:35:56.901072 13118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41713
--ranger_config_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:56.902127 13118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:56.903223 13118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:56.911805 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:56.911808 13118 server_base.cc:1034] running on GCE node
W20260927 16:35:56.911860 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:56.911835 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:56.912636 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:56.913699 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:56.914885 13118 hybrid_clock.cc:648] HybridClock initialized: now 1790526956914854 us; error 52 us; skew 500 ppm
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:56.917784 13118 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:35:56.918725 13118 webserver.cc:460] Webserver started at http://127.0.98.62:40753/ using document root <none> and password file <none>
I20260927 16:35:56.919291 13118 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:56.919389 13118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:35:56 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:56.975838 13118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:35:56.978086 13134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:56.979436 13118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:35:56.979565 13118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8f0fbe408dd249c084a49bc2d3a350fb"
format_stamp: "Formatted at 2026-09-27 16:35:54 on dist-test-slave-t27c"
server_key: "45cdeae6e0d5b2de6172f3e5a66fbfe6"
server_key_iv: "620ac0477eac339314a70c265aa4b80b"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:56.980103 13118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:56.995096 13118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:56.997778 13118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:56.997977 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:57.005065 13186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41713 every 8 connection(s)
I20260927 16:35:57.005066 13118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41713
I20260927 16:35:57.006151 13118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:35:57.007123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13118
I20260927 16:35:57.007567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37331
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41713
--webserver_port=42955
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:35:57.010587 13187 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:35:57.012274 13187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Bootstrap starting.
I20260927 16:35:57.028429 13187 log.cc:824] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:57.034794 13187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:35:57.035180 13187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Bootstrap complete.
I20260927 16:35:57.039227 13187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:57.039661 13187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0fbe408dd249c084a49bc2d3a350fb, State: Initialized, Role: FOLLOWER
I20260927 16:35:57.040129 13187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:57.040275 13187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:35:57.040366 13187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:35:57.040484 13187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:35:57.042352 13187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:57.042672 13187 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0fbe408dd249c084a49bc2d3a350fb; no voters:
I20260927 16:35:57.042969 13187 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:35:57.043097 13191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:35:57.043345 13191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [term 2 LEADER]: Becoming Leader. State: Replica: 8f0fbe408dd249c084a49bc2d3a350fb, State: Running, Role: LEADER
I20260927 16:35:57.043712 13191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260927 16:35:57.044134 13187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:35:57.045148 13193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0fbe408dd249c084a49bc2d3a350fb. Latest consensus state: current_term: 2 leader_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260927 16:35:57.045277 13193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:57.045607 13200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:35:57.045596 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fbe408dd249c084a49bc2d3a350fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260927 16:35:57.045760 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb [sys.catalog]: This master's current role is: LEADER
I20260927 16:35:57.048758 13200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:35:57.049352 13187 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:35:57.049417 13200 catalog_manager.cc:1259] Loaded cluster ID: c5986065109644598e72f0eceb47bc13
I20260927 16:35:57.049464 13200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:35:57.052081 13200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:35:57.052788 13200 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Loaded TSK: 0
I20260927 16:35:57.053984 13200 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:35:57.145066 13189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:57.145439 13189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:57.145588 13189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:57.150128 13189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:57.150318 13189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:57.150453 13189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:35:57.156332 13189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42955
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:57.157371 13189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:57.158452 13189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:57.170110 13227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:57.170449 13189 server_base.cc:1034] running on GCE node
W20260927 16:35:57.170615 13224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:57.171322 13225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:57.171731 13189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:57.172520 13189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:57.173707 13189 hybrid_clock.cc:648] HybridClock initialized: now 1790526957173675 us; error 47 us; skew 500 ppm
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:57.177363 13189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:35:57.178473 13189 webserver.cc:460] Webserver started at http://127.0.98.1:42955/ using document root <none> and password file <none>
I20260927 16:35:57.179052 13189 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:57.179158 13189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:57.211947 13189 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.003s
I20260927 16:35:57.219019 13235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.220713 13189 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260927 16:35:57.220861 13189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "9137525e351f4ea7857b20e39ad9f741"
format_stamp: "Formatted at 2026-09-27 16:35:55 on dist-test-slave-t27c"
server_key: "1899e879d3bb78e5a5dd8cf7f5b12f9c"
server_key_iv: "bb290f1bd56aa50adbd24725b3b9c6fd"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:57.221745 13189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:57.257429 13189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:57.260967 13189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:57.261194 13189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:57.261873 13189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:57.262933 13189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:57.263001 13189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.263077 13189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:57.263111 13189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.275800 13189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37331
I20260927 16:35:57.275950 13348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37331 every 8 connection(s)
I20260927 16:35:57.277206 13189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:35:57.286602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13189
I20260927 16:35:57.287102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33637
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41713
--webserver_port=43843
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:57.305116 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.279762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52271 (local address 127.0.98.62:41713)
0927 16:35:57.280454 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:57.280478 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:57.286349 (+ 5871us) server_negotiation.cc:407] Connection header received
0927 16:35:57.287904 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:57.287919 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:57.288373 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:57.288736 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:57.290499 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.292092 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.293879 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.294644 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.294805 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.298296 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:57.298305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:57.298323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:57.298356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:57.300588 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.301401 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.301411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.301419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.301588 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.302115 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.302121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.302126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.302737 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:57.302914 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:57.303780 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 16:35:57.304092 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":120,"threads_started":1}
I20260927 16:35:57.305130 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.279863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.1:52271)
0927 16:35:57.280758 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:57.280815 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:57.287234 (+ 6419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:57.289245 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:57.289254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:57.289619 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:57.290292 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.290313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.292249 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.292254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.293735 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.293746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.295048 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.295052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.295246 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.295987 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:57.296009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:57.298018 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:57.300729 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.300738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.300760 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.301226 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.301737 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.301743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.301748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.301972 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.302977 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:57.302986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:57.303387 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 16:35:57.303621 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 16:35:57.304028 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":138,"threads_started":1}
I20260927 16:35:57.308128 13349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:57.308399 13349 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.308971 13349 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.311540 13349 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41713 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:35:57.311662 13349 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.311844 13349 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:57.312781 13349 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.312945 13349 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.313769 13349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:35:57.431073 13353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:57.431365 13353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:57.431478 13353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:57.436105 13353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:57.436192 13353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:57.436303 13353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:35:57.441746 13353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43843
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:57.442888 13353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:57.444108 13353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:57.454047 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:57.454196 13353 server_base.cc:1034] running on GCE node
W20260927 16:35:57.454367 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:57.454398 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:57.455178 13353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:57.455891 13353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:57.457105 13353 hybrid_clock.cc:648] HybridClock initialized: now 1790526957457076 us; error 50 us; skew 500 ppm
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:57.459976 13353 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:35:57.461002 13353 webserver.cc:460] Webserver started at http://127.0.98.2:43843/ using document root <none> and password file <none>
I20260927 16:35:57.461565 13353 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:57.461634 13353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:57.479641 13353 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:35:57.482885 13369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.484361 13353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:35:57.484503 13353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "5be0f07bbf874e56800b8b8b62050997"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "6a46bf6e8f2f78b55805df98938fb663"
server_key_iv: "6e68d6a3c51e620c4036c524e4a6e07c"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:57.485143 13353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:57.519768 13353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:57.522774 13353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:57.522992 13353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:57.523584 13353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:57.524618 13353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:57.524724 13353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.524817 13353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:57.524873 13353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.535918 13353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33637
I20260927 16:35:57.536156 13482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33637 every 8 connection(s)
I20260927 16:35:57.537245 13353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:35:57.541112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13353
I20260927 16:35:57.541601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38947
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41713
--webserver_port=38365
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:57.555673 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.540996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52577 (local address 127.0.98.62:41713)
0927 16:35:57.542938 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:57.542946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:57.542965 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:35:57.543018 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:57.543022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:57.543078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:57.543179 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:57.544769 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.545653 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.547586 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.547942 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.548024 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.551300 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:57.551304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:57.551307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:57.551338 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:57.553136 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.553754 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.553757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.553759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.553865 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.554335 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.554339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.554341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.554566 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:57.554654 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:57.555364 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 16:35:57.555495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831}
I20260927 16:35:57.557523 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.539944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.2:52577)
0927 16:35:57.541331 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:57.541391 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:57.542485 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:57.543453 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:57.543465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:57.543863 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:57.544578 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.544601 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.545982 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.545986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.547440 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.547451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.548287 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.548292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.548485 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.549201 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:57.549225 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:57.551095 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:57.553277 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.553283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.553300 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.553630 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.553989 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.553992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.553994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.554219 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.554722 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:57.554729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:57.555071 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 16:35:57.556145 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0927 16:35:57.556501 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":141,"threads_started":1}
I20260927 16:35:57.559156 13483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:57.559399 13483 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.559988 13483 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.561755 13483 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41713 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:35:57.561896 13483 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.562060 13483 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:57.562927 13483 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.563349 13483 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.564152 13483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:35:57.685213 13486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:35:57.685542 13486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:35:57.685622 13486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:35:57.691165 13486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:35:57.691247 13486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:35:57.691335 13486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:35:57.698222 13486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38365
--tserver_master_addrs=127.0.98.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:35:57.699759 13486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:35:57.700906 13486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:35:57.710415 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:57.710407 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:35:57.710839 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:35:57.711201 13486 server_base.cc:1034] running on GCE node
I20260927 16:35:57.711650 13486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:35:57.712281 13486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:35:57.715592 13486 hybrid_clock.cc:648] HybridClock initialized: now 1790526957715547 us; error 47 us; skew 500 ppm
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:57.719197 13486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:35:57.720219 13486 webserver.cc:460] Webserver started at http://127.0.98.3:38365/ using document root <none> and password file <none>
I20260927 16:35:57.720906 13486 fs_manager.cc:359] Metadata directory not provided
I20260927 16:35:57.721022 13486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:57.739225 13486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 16:35:57.742347 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.744074 13486 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:35:57.744215 13486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "416bfd42868c4863bf6a8d2e2c6c6a06"
format_stamp: "Formatted at 2026-09-27 16:35:56 on dist-test-slave-t27c"
server_key: "9aa5b4c250b65080557244550f249ba4"
server_key_iv: "d909bbda3792c78285a42c97824a8ac5"
server_key_version: "kuduclusterkey@0"
I20260927 16:35:57.744961 13486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:35:57.769405 13486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:35:57.773278 13486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:35:57.773509 13486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:35:57.774379 13486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:35:57.775527 13486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:35:57.775640 13486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.775727 13486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:35:57.775780 13486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:35:57.787320 13486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38947
I20260927 16:35:57.787533 13616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38947 every 8 connection(s)
I20260927 16:35:57.788461 13486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:35:57.792114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13486
Sep 27 16:35:57 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:35:57.804724 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.793076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.0.98.62:41713)
0927 16:35:57.793781 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:57.793791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:57.793802 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:35:57.793835 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:57.793838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:57.793872 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:57.793947 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:57.795173 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.796618 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.797807 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.798112 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.798228 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.800216 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:57.800223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:57.800229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:57.800261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:57.802782 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.803319 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.803323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.803325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.803410 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.803718 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.803724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.803726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.803921 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:57.804156 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:57.804284 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 16:35:57.804412 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":627,"thread_start_us":92,"threads_started":1}
I20260927 16:35:57.805685 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.790469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55779 (local address 127.0.98.62:41713)
0927 16:35:57.790602 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:57.790607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:57.791578 (+ 971us) server_negotiation.cc:407] Connection header received
0927 16:35:57.793605 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:57.793609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:57.793666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:57.793743 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:57.795356 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.795901 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.797574 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.797901 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.797967 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.801208 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:57.801215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:57.801221 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:57.801278 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:57.803017 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.803607 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.803610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.803612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.803685 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:57.804087 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:57.804091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:57.804093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:57.804322 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:57.804401 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:57.805148 (+ 747us) server_negotiation.cc:299] Negotiation successful
0927 16:35:57.805269 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":54}
I20260927 16:35:57.806588 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:57.790946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41713 (local address 127.0.98.3:55779)
0927 16:35:57.791418 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:57.791472 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:57.793392 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:57.794054 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:57.794065 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:57.794482 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:35:57.795209 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.795229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.796033 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.796036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.797439 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:57.797449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:57.798196 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:57.798204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:57.798404 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:57.799086 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:35:57.799111 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:35:57.801000 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:35:57.803126 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.803131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.803148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.803516 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.803770 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:35:57.803774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:35:57.803776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:35:57.803959 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:35:57.804489 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:35:57.804495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:35:57.804857 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 16:35:57.805095 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 16:35:57.805451 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20260927 16:35:57.807762 13617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41713
I20260927 16:35:57.808012 13617 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.808509 13617 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.810073 13617 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41713 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:35:57.810245 13617 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.810420 13617 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:57.811225 13617 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:57.811396 13617 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
W20260927 16:35:57.812147 13617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 16:35:58 dist-test-slave-t27c krb5kdc[12155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:35:58.314002 13349 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:58.314344 13349 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:58.315543 13151 ts_manager.cc:194] Registered new tserver with Master: 9137525e351f4ea7857b20e39ad9f741 (127.0.98.1:37331)
I20260927 16:35:58.317298 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52271
I20260927 16:35:58.564380 13483 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:58.564679 13483 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:58.565521 13151 ts_manager.cc:194] Registered new tserver with Master: 5be0f07bbf874e56800b8b8b62050997 (127.0.98.2:33637)
I20260927 16:35:58.566372 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52577
Sep 27 16:35:58 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:35:58 dist-test-slave-t27c krb5kdc[12155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:35:58.812367 13617 heartbeater.cc:464] Registering TS with master...
I20260927 16:35:58.812693 13617 heartbeater.cc:511] Master 127.0.98.62:41713 requested a full tablet report, sending...
I20260927 16:35:58.813381 13151 ts_manager.cc:194] Registered new tserver with Master: 416bfd42868c4863bf6a8d2e2c6c6a06 (127.0.98.3:38947)
I20260927 16:35:58.814088 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55779
I20260927 16:35:58.824551 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:35:58.855098 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.827493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.98.62:41713)
0927 16:35:58.827742 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:58.827747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:58.844272 (+ 16525us) server_negotiation.cc:407] Connection header received
0927 16:35:58.844560 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:58.844565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:58.844632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:58.844713 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:35:58.845919 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.846778 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.847928 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.848306 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.848387 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.850194 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:35:58.850199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:35:58.850203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:35:58.850233 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:35:58.853079 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:58.853590 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:58.853594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:58.853596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:58.853687 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:35:58.853989 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:35:58.853992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:35:58.853994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:35:58.854248 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:35:58.854356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:58.854726 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 16:35:58.854914 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260927 16:35:58.859972 13151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:35:58.865307 13151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:35:58.902951 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.890812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33637 (local address 127.0.0.1:44490)
0927 16:35:58.895145 (+ 4333us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:58.895210 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:58.895335 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:58.896362 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:58.896369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:58.896391 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:58.896522 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.896528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.898776 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.898779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.900162 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.900850 (+ 688us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.902413 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.902416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.902531 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.902567 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 16:35:58.902674 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 16:35:58.902791 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3424,"thread_start_us":113,"threads_started":1}
I20260927 16:35:58.904729 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.891574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.0.98.2:33637)
0927 16:35:58.895632 (+ 4058us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:58.895640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:58.895663 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:35:58.895738 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:58.895744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:58.895989 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:58.896179 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:58.896716 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.898640 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.901235 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.902076 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.902205 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.902353 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:58.903814 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0927 16:35:58.903914 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3956,"thread_start_us":88,"threads_started":1}
I20260927 16:35:58.908169 13417 tablet_service.cc:1467] Processing CreateTablet for tablet 027f622db1f6415491edeff7959fcaca (DEFAULT_TABLE table=test-table [id=db854ed8be054b5883be60ea2c62cf45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:58.909364 13417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f622db1f6415491edeff7959fcaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:58.909562 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.893807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37331 (local address 127.0.0.1:41384)
0927 16:35:58.903556 (+ 9749us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:58.903569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:58.903636 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:58.905152 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:58.905155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:58.905172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:58.905249 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.905254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.906629 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.906632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.907690 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.907700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.909178 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.909182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.909273 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.909298 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:35:58.909369 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:35:58.909440 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9696,"thread_start_us":75,"threads_started":1}
I20260927 16:35:58.909998 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.893983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.98.1:37331)
0927 16:35:58.900948 (+ 6965us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:58.900956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:58.904494 (+ 3538us) server_negotiation.cc:407] Connection header received
0927 16:35:58.904589 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:58.904595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:58.904813 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:58.904973 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:58.905378 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.906494 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.907831 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.908841 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.908944 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.909105 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:58.909769 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 16:35:58.909869 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6848,"thread_start_us":88,"threads_started":1}
I20260927 16:35:58.911576 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.893452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38947 (local address 127.0.0.1:35500)
0927 16:35:58.901044 (+ 7592us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:58.901057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:58.901124 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:58.906683 (+ 5559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:58.906686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:58.906704 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:58.906796 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.906801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.908230 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.908234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.910098 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:58.910109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.911217 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.911221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:58.911309 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.911332 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:35:58.911380 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 16:35:58.911443 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7531,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 16:35:58.912269 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.893613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.0.98.3:38947)
0927 16:35:58.906015 (+ 12402us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:58.906022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:58.906047 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:35:58.906123 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:58.906128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:58.906368 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:58.906514 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:58.906922 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.908084 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.910276 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:58.911063 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:58.911831 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:58.911991 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:58.912040 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:35:58.912127 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12288,"thread_start_us":97,"threads_started":1}
I20260927 16:35:58.912933 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 027f622db1f6415491edeff7959fcaca (DEFAULT_TABLE table=test-table [id=db854ed8be054b5883be60ea2c62cf45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:58.913946 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f622db1f6415491edeff7959fcaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:58.924031 13543 tablet_service.cc:1467] Processing CreateTablet for tablet 027f622db1f6415491edeff7959fcaca (DEFAULT_TABLE table=test-table [id=db854ed8be054b5883be60ea2c62cf45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:35:58.925135 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f622db1f6415491edeff7959fcaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:35:58.926940 13654 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Bootstrap starting.
I20260927 16:35:58.930595 13653 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Bootstrap starting.
I20260927 16:35:58.931715 13654 tablet_bootstrap.cc:654] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:58.932605 13654 log.cc:824] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:58.932821 13653 tablet_bootstrap.cc:654] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:58.933634 13653 log.cc:824] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:58.943219 13657 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Bootstrap starting.
I20260927 16:35:58.944716 13653 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: No bootstrap required, opened a new log
I20260927 16:35:58.944715 13654 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: No bootstrap required, opened a new log
I20260927 16:35:58.944957 13654 ts_tablet_manager.cc:1397] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260927 16:35:58.944990 13653 ts_tablet_manager.cc:1397] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260927 16:35:58.945722 13657 tablet_bootstrap.cc:654] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Neither blocks nor log segments found. Creating new log.
I20260927 16:35:58.946913 13657 log.cc:824] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Log is configured to *not* fsync() on all Append() calls
I20260927 16:35:58.949572 13653 raft_consensus.cc:348] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.950062 13653 raft_consensus.cc:374] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:58.950188 13653 raft_consensus.cc:729] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be0f07bbf874e56800b8b8b62050997, State: Initialized, Role: FOLLOWER
I20260927 16:35:58.950807 13653 consensus_queue.cc:260] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.952133 13653 ts_tablet_manager.cc:1428] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 16:35:58.952553 13483 heartbeater.cc:503] Master 127.0.98.62:41713 was elected leader, sending a full tablet report...
I20260927 16:35:58.955667 13654 raft_consensus.cc:348] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.955956 13654 raft_consensus.cc:374] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:58.956008 13654 raft_consensus.cc:729] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9137525e351f4ea7857b20e39ad9f741, State: Initialized, Role: FOLLOWER
I20260927 16:35:58.957947 13654 consensus_queue.cc:260] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.961575 13654 ts_tablet_manager.cc:1428] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260927 16:35:58.962752 13349 heartbeater.cc:503] Master 127.0.98.62:41713 was elected leader, sending a full tablet report...
I20260927 16:35:58.968425 13657 tablet_bootstrap.cc:492] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: No bootstrap required, opened a new log
I20260927 16:35:58.968650 13657 ts_tablet_manager.cc:1397] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260927 16:35:58.973860 13657 raft_consensus.cc:348] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.974258 13657 raft_consensus.cc:374] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:35:58.974357 13657 raft_consensus.cc:729] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416bfd42868c4863bf6a8d2e2c6c6a06, State: Initialized, Role: FOLLOWER
I20260927 16:35:58.974947 13657 consensus_queue.cc:260] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.976102 13657 ts_tablet_manager.cc:1428] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 16:35:58.976537 13617 heartbeater.cc:503] Master 127.0.98.62:41713 was elected leader, sending a full tablet report...
I20260927 16:35:58.996763 13660 raft_consensus.cc:482] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:35:58.997025 13660 raft_consensus.cc:504] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:58.998509 13660 leader_election.cc:288] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5be0f07bbf874e56800b8b8b62050997 (127.0.98.2:33637), 416bfd42868c4863bf6a8d2e2c6c6a06 (127.0.98.3:38947)
I20260927 16:35:59.005414 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.999924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54773 (local address 127.0.98.2:33637)
0927 16:35:59.000199 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:59.000203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:59.000569 (+ 366us) server_negotiation.cc:407] Connection header received
0927 16:35:59.000796 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:59.000800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:59.000857 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:59.000910 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:59.001733 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.002503 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.004108 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.004730 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.004813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:59.004882 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:59.005218 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 16:35:59.005291 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260927 16:35:59.005981 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:59.000062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60771 (local address 127.0.98.3:38947)
0927 16:35:59.000295 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:59.000300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:59.001587 (+ 1287us) server_negotiation.cc:407] Connection header received
0927 16:35:59.001705 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:59.001708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:59.001763 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:59.001849 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:35:59.002250 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.003051 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.004576 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.005257 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.005336 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:59.005397 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:59.005783 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 16:35:59.005860 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 16:35:59.006031 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.998970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33637 (local address 127.0.98.1:54773)
0927 16:35:59.000475 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:59.000490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:59.000649 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:59.001064 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:59.001067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:59.001089 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:59.001239 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:59.001246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.002652 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.002655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:59.003961 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:59.003973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.004986 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.004989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:59.005082 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:59.005105 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:35:59.005796 (+ 691us) client_negotiation.cc:240] Negotiation successful
0927 16:35:59.005875 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":121,"threads_started":1}
I20260927 16:35:59.006414 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:58.999400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38947 (local address 127.0.98.1:60771)
0927 16:35:59.001490 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:59.001505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:35:59.001605 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:35:59.001955 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:35:59.001958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:35:59.001976 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:35:59.002093 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:59.002099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.003196 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.003199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:59.004433 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:35:59.004443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.005523 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.005527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:35:59.005629 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:59.005649 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:35:59.006139 (+ 490us) client_negotiation.cc:240] Negotiation successful
0927 16:35:59.006251 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"thread_start_us":106,"threads_started":1}
I20260927 16:35:59.007045 13437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f622db1f6415491edeff7959fcaca" candidate_uuid: "9137525e351f4ea7857b20e39ad9f741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be0f07bbf874e56800b8b8b62050997" is_pre_election: true
I20260927 16:35:59.007169 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f622db1f6415491edeff7959fcaca" candidate_uuid: "9137525e351f4ea7857b20e39ad9f741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" is_pre_election: true
I20260927 16:35:59.007373 13437 raft_consensus.cc:2454] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9137525e351f4ea7857b20e39ad9f741 in term 0.
I20260927 16:35:59.007427 13554 raft_consensus.cc:2454] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9137525e351f4ea7857b20e39ad9f741 in term 0.
I20260927 16:35:59.007889 13239 leader_election.cc:302] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5be0f07bbf874e56800b8b8b62050997, 9137525e351f4ea7857b20e39ad9f741; no voters:
I20260927 16:35:59.008136 13660 raft_consensus.cc:2781] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:35:59.008219 13660 raft_consensus.cc:482] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:35:59.008255 13660 raft_consensus.cc:3037] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:59.009582 13660 raft_consensus.cc:504] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:59.009931 13660 leader_election.cc:288] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [CANDIDATE]: Term 1 election: Requested vote from peers 5be0f07bbf874e56800b8b8b62050997 (127.0.98.2:33637), 416bfd42868c4863bf6a8d2e2c6c6a06 (127.0.98.3:38947)
I20260927 16:35:59.010303 13437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f622db1f6415491edeff7959fcaca" candidate_uuid: "9137525e351f4ea7857b20e39ad9f741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be0f07bbf874e56800b8b8b62050997"
I20260927 16:35:59.010373 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f622db1f6415491edeff7959fcaca" candidate_uuid: "9137525e351f4ea7857b20e39ad9f741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06"
I20260927 16:35:59.010423 13437 raft_consensus.cc:3037] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:59.010485 13554 raft_consensus.cc:3037] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:35:59.011636 13554 raft_consensus.cc:2454] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9137525e351f4ea7857b20e39ad9f741 in term 1.
I20260927 16:35:59.011667 13437 raft_consensus.cc:2454] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9137525e351f4ea7857b20e39ad9f741 in term 1.
I20260927 16:35:59.011993 13236 leader_election.cc:302] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416bfd42868c4863bf6a8d2e2c6c6a06, 9137525e351f4ea7857b20e39ad9f741; no voters:
I20260927 16:35:59.012225 13660 raft_consensus.cc:2781] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:35:59.012518 13660 raft_consensus.cc:686] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [term 1 LEADER]: Becoming Leader. State: Replica: 9137525e351f4ea7857b20e39ad9f741, State: Running, Role: LEADER
I20260927 16:35:59.012913 13660 consensus_queue.cc:237] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } }
I20260927 16:35:59.016150 13151 catalog_manager.cc:5495] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9137525e351f4ea7857b20e39ad9f741 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9137525e351f4ea7857b20e39ad9f741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9137525e351f4ea7857b20e39ad9f741" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } health_report { overall_health: UNKNOWN } } }
W20260927 16:35:59.029955 13350 tablet.cc:2369] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:35:59.041296 13618 tablet.cc:2369] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:35:59.054956 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:59.049987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.98.1:37331)
0927 16:35:59.050123 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:35:59.050128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:35:59.050145 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:35:59.050302 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:35:59.050305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:35:59.050406 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:35:59.050532 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:35:59.050869 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.051661 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.052875 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:35:59.053223 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:35:59.053337 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:35:59.053691 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:35:59.054279 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:35:59.054400 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:35:59.054733 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 16:35:59.054794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 16:35:59.056742 13484 tablet.cc:2369] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:35:59.062151 13554 raft_consensus.cc:1261] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 1 FOLLOWER]: Refusing update from remote peer 9137525e351f4ea7857b20e39ad9f741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:35:59.062229 13437 raft_consensus.cc:1261] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 1 FOLLOWER]: Refusing update from remote peer 9137525e351f4ea7857b20e39ad9f741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:35:59.062956 13660 consensus_queue.cc:1035] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416bfd42868c4863bf6a8d2e2c6c6a06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:35:59.063206 13664 consensus_queue.cc:1035] T 027f622db1f6415491edeff7959fcaca P 9137525e351f4ea7857b20e39ad9f741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be0f07bbf874e56800b8b8b62050997" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:35:59.074879 13667 mvcc.cc:205] Tried to move back new op lower bound from 7333998424310693888 to 7333998424119996416. Current Snapshot: MvccSnapshot[applied={T|T < 7333998424310693888}]
I20260927 16:35:59.080451 13669 mvcc.cc:205] Tried to move back new op lower bound from 7333998424310693888 to 7333998424119996416. Current Snapshot: MvccSnapshot[applied={T|T < 7333998424310693888}]
I20260927 16:35:59.086764 13150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:35:59.086936 13150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:35:59.089557 13150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f0fbe408dd249c084a49bc2d3a350fb: Sending DeleteTablet for 3 replicas of tablet 027f622db1f6415491edeff7959fcaca
I20260927 16:35:59.090855 13417 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f622db1f6415491edeff7959fcaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44490
I20260927 16:35:59.091777 13543 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f622db1f6415491edeff7959fcaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35500
I20260927 16:35:59.092660 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f622db1f6415491edeff7959fcaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:35:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41384
I20260927 16:35:59.093815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13189
I20260927 16:35:59.098387 13677 tablet_replica.cc:330] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: stopping tablet replica
I20260927 16:35:59.098868 13677 raft_consensus.cc:2229] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:35:59.099184 13676 tablet_replica.cc:330] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997: stopping tablet replica
I20260927 16:35:59.102545 13677 raft_consensus.cc:2258] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:35:59.103004 13676 raft_consensus.cc:2229] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:35:59.103579 13676 raft_consensus.cc:2258] T 027f622db1f6415491edeff7959fcaca P 5be0f07bbf874e56800b8b8b62050997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:35:59.105263 13677 ts_tablet_manager.cc:1905] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:35:59.105993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13353
W20260927 16:35:59.105985 13138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37331 (error 108)
I20260927 16:35:59.107780 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:59.107423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37331 (local address 127.0.0.1:41406)
0927 16:35:59.107606 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:59.107673 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 16:35:59.108157 13138 catalog_manager.cc:4584] TS 9137525e351f4ea7857b20e39ad9f741 (127.0.98.1:37331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 027f622db1f6415491edeff7959fcaca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37331: connect: Connection refused (error 111)
I20260927 16:35:59.115623 13677 ts_tablet_manager.cc:1918] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:35:59.115818 13677 log.cc:1196] T 027f622db1f6415491edeff7959fcaca P 416bfd42868c4863bf6a8d2e2c6c6a06: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/027f622db1f6415491edeff7959fcaca
I20260927 16:35:59.115913 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:35:59.115620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33637 (local address 127.0.0.1:44498)
0927 16:35:59.115750 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:35:59.115810 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 16:35:59.116113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13486
I20260927 16:35:59.123469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13118
I20260927 16:35:59.130091 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:35:59.466312 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:35:59.466414 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:35:59.811134 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:35:59.811 UTC [12185] LOG: received smart shutdown request
2026-09-27 16:35:59.815 UTC [12185] LOG: background worker "logical replication launcher" (PID 12194) exited with exit code 1
2026-09-27 16:35:59.815 UTC [12189] LOG: shutting down
2026-09-27 16:35:59.966 UTC [12185] LOG: database system is shut down
2026-09-27T16:35:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48224 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:36:00 dist-test-slave-t27c krb5kdc[13695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:36:00 dist-test-slave-t27c krb5kdc[13695](info): set up 2 sockets
Sep 27 16:36:00 dist-test-slave-t27c krb5kdc[13695](info): commencing operation
krb5kdc: starting...
W20260927 16:36:01.146473 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:36:01.273264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:01.384719 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:01.385036 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:01.385106 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:01.389418 13708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:01.389503 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:01.389540 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:01.389593 13708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:01.389631 13708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:01.395062 13708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:01.396188 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:01.397212 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:01.406301 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:01.406247 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:01.406366 13708 server_base.cc:1034] running on GCE node
W20260927 16:36:01.406313 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:01.407076 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:01.408222 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:01.409421 13708 hybrid_clock.cc:648] HybridClock initialized: now 1790526961409386 us; error 50 us; skew 500 ppm
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:01.412355 13708 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:01.413231 13708 webserver.cc:460] Webserver started at http://127.0.98.62:40875/ using document root <none> and password file <none>
I20260927 16:36:01.413806 13708 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:01.413897 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:01.414129 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:01.416173 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "dbb3fc0ed1d442f192c04e46809cc95b"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "99c12504880c11736d11c1e77eb43bf9"
server_key_iv: "69f70bed993dcde63b8175c1b126cfe0"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.416750 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "dbb3fc0ed1d442f192c04e46809cc95b"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "99c12504880c11736d11c1e77eb43bf9"
server_key_iv: "69f70bed993dcde63b8175c1b126cfe0"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.420573 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 16:36:01.423015 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:01.424263 13708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:36:01.424418 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "dbb3fc0ed1d442f192c04e46809cc95b"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "99c12504880c11736d11c1e77eb43bf9"
server_key_iv: "69f70bed993dcde63b8175c1b126cfe0"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.424535 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:01.437813 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:01.441324 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:01.441534 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:01.448566 13708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46113
I20260927 16:36:01.448662 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46113 every 8 connection(s)
I20260927 16:36:01.449654 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:01.452440 13776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:01.456270 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.459471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13708
I20260927 16:36:01.459584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:36:01.459882 392 external_mini_cluster.cc:1428] Setting key b3eb0f2ea2263b59473bebcd549e11d3
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 16:36:01.471855 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.458551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.0.98.62:46113)
0927 16:36:01.459019 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.459037 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.459653 (+ 616us) server_negotiation.cc:407] Connection header received
0927 16:36:01.460430 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.460434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.460610 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.460804 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.461642 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.462568 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.463616 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.463978 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.464158 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.466618 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.466623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.466634 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.466655 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.468502 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.469049 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.469062 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.469064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.469222 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.469558 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.469563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.469565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.469952 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.470100 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.470495 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.470789 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260927 16:36:01.471855 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.458551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.0.1:39446)
0927 16:36:01.459130 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.459170 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.460106 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.460807 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.460813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.461250 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.461492 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.461505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.462691 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.462695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.463498 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.463508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.464133 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.464137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.464318 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.464942 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.464960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.466457 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.468641 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.468649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.468660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.468935 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.469323 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.469328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.469330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.469480 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.470070 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.470076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.470247 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.470435 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.470838 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":189,"spinlock_wait_cycles":37120,"thread_start_us":209,"threads_started":1}
I20260927 16:36:01.474716 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.475452 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.475186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35616)
0927 16:36:01.475307 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.475389 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 16:36:01.475634 13725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
I20260927 16:36:01.476421 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.476222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35632)
0927 16:36:01.476323 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.476373 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:36:01.476577 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:36:01.485029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46239
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:01.525838 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } attempt: 1
I20260927 16:36:01.526917 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.526584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35648)
0927 16:36:01.526756 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.526820 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 16:36:01.527779 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.527559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35650)
0927 16:36:01.527684 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.527730 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 16:36:01.527997 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
I20260927 16:36:01.596165 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } attempt: 2
W20260927 16:36:01.596154 13783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:01.596565 13783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:01.596657 13783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:36:01.597065 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.596760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35658)
0927 16:36:01.596925 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.596990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 16:36:01.597750 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.597558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35666)
0927 16:36:01.597656 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.597701 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:36:01.597892 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46239: connect: Connection refused (error 111)
W20260927 16:36:01.601092 13783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:01.601147 13783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:01.601167 13783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:01.601183 13783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:01.601199 13783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:01.606874 13783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:01.608245 13783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:01.609485 13783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:01.617856 13788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:01.617846 13789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:01.617846 13791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:01.618045 13783 server_base.cc:1034] running on GCE node
I20260927 16:36:01.618769 13783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:01.619886 13783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:01.621076 13783 hybrid_clock.cc:648] HybridClock initialized: now 1790526961621056 us; error 46 us; skew 500 ppm
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:01.623996 13783 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:36:01.624821 13783 webserver.cc:460] Webserver started at http://127.0.98.61:42143/ using document root <none> and password file <none>
I20260927 16:36:01.625326 13783 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:01.625383 13783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:01.625543 13783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:01.627668 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data/instance:
uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "60012c470de56bf15f35072c28d992fb"
server_key_iv: "91e6847649bde0fb914a74ad8b897aca"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.628188 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal/instance:
uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "60012c470de56bf15f35072c28d992fb"
server_key_iv: "91e6847649bde0fb914a74ad8b897aca"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.632066 13783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:36:01.634456 13798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:01.635663 13783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:01.635764 13783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "60012c470de56bf15f35072c28d992fb"
server_key_iv: "91e6847649bde0fb914a74ad8b897aca"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.635847 13783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:01.650527 13783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:01.653424 13783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:01.653591 13783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:01.660643 13783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46239
I20260927 16:36:01.660671 13850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46239 every 8 connection(s)
I20260927 16:36:01.661628 13783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:36:01.664345 13851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:01.668085 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.671265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13783
I20260927 16:36:01.671391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-1/wal/instance
I20260927 16:36:01.671679 392 external_mini_cluster.cc:1428] Setting key 4a2b066d27cf41db751f2d0602f3b8d1
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 16:36:01.683194 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.669966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.0.98.62:46113)
0927 16:36:01.670117 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.670121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.670980 (+ 859us) server_negotiation.cc:407] Connection header received
0927 16:36:01.672244 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.672247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.672297 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.672379 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.673949 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.674779 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.675939 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.676186 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.676308 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.678933 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.678936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.678938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.678965 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.680634 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.681241 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.681245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.681247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.681333 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.681767 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.681770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.681772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.681974 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.682109 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.682777 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.682936 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:36:01.684262 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.670377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.0.1:39462)
0927 16:36:01.670825 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.670872 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.671976 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.672603 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.672614 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.673134 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.673782 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.673796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.674907 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.674910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.675801 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.675810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.676352 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.676356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.676538 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.677212 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.677229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.678735 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.680771 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.680776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.680794 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.681134 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.681430 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.681434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.681436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.681656 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.682131 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.682140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.682472 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.682701 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.683049 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260927 16:36:01.685881 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.689085 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } attempt: 3
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:36:01.699810 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.689504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35686)
0927 16:36:01.689692 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.689709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.689786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.690059 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.690062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.690287 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.690859 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.690879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.692011 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.692015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.692716 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.692726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.693253 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.693255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.693336 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.693813 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.693829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.695290 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.697968 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.697971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.697972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.698212 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.698596 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.698599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.698676 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.699242 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.699246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.699352 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.699474 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.699622 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 16:36:01.699817 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.686448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35678)
0927 16:36:01.686550 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.686564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.686641 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.687449 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.687451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.687589 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.687679 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.687684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.689300 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.689303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.690235 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.690246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.691078 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.691081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.691257 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.691774 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.691786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.694777 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.697962 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.697965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.697967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.698151 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.698557 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.698561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.698647 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.699245 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.699247 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.699340 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.699425 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.699625 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":63}
I20260927 16:36:01.699937 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.689553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.98.61:46239)
0927 16:36:01.689852 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.689863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.689874 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:36:01.689905 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.689907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.689950 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.690043 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.691014 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.691891 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.692843 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.693151 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.693233 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.695464 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.695468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.695472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.695499 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.697846 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698348 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.698355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.698359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.698475 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698798 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.698800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.698801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.699134 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.699274 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.699586 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.699731 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":226,"spinlock_wait_cycles":9472,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 16:36:01.700649 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.686593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.0.98.61:46239)
0927 16:36:01.686979 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.686991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.687019 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:36:01.687068 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.687075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.687302 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.687460 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.687796 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.689172 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.690352 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.690985 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.691072 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.694922 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.694931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.694944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.694965 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.697863 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698302 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.698310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.698332 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.698465 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.698748 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.698753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.698755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.699134 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.699275 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.699593 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.699799 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":263,"spinlock_wait_cycles":3840,"thread_start_us":140,"threads_started":1}
I20260927 16:36:01.701907 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.702021 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.702466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44907
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:01.702718 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.702410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43088)
0927 16:36:01.702596 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.702648 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 16:36:01.702839 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.702521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43104)
0927 16:36:01.702672 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.702756 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 16:36:01.703020 13800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.703441 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.703169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43118)
0927 16:36:01.703337 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.703388 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 16:36:01.703570 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.704066 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.703860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43120)
0927 16:36:01.703968 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.704012 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:36:01.704211 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.734731 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 1
I20260927 16:36:01.735646 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.735320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43132)
0927 16:36:01.735500 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.735566 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 16:36:01.736372 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.736163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43134)
0927 16:36:01.736272 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.736322 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:36:01.736497 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.753480 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 1
I20260927 16:36:01.754374 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.754037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43140)
0927 16:36:01.754227 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.754289 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 16:36:01.755084 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.754873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43144)
0927 16:36:01.754979 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.755032 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:36:01.755210 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.811666 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 2
I20260927 16:36:01.812476 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.812180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43158)
0927 16:36:01.812323 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.812399 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 16:36:01.813161 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.812962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43168)
0927 16:36:01.813063 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.813115 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:36:01.813287 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
W20260927 16:36:01.813954 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:01.814265 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:01.814359 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:01.818333 13859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:01.818416 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:01.818454 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:01.818507 13859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:01.818547 13859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:01.823365 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 2
I20260927 16:36:01.823953 13859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46113,127.0.98.61:46239,127.0.98.60:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:01.824142 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.823858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43174)
0927 16:36:01.823993 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.824049 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 16:36:01.824802 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.824623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43182)
0927 16:36:01.824714 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.824755 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:36:01.824935 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44907: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44907: connect: Connection refused (error 111)
I20260927 16:36:01.825284 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:01.826349 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:01.834685 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:01.834678 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:01.834682 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:01.834993 13859 server_base.cc:1034] running on GCE node
I20260927 16:36:01.835508 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:01.836591 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:01.837771 13859 hybrid_clock.cc:648] HybridClock initialized: now 1790526961837755 us; error 27 us; skew 500 ppm
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:01.840694 13859 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:36:01.841548 13859 webserver.cc:460] Webserver started at http://127.0.98.60:39529/ using document root <none> and password file <none>
I20260927 16:36:01.842098 13859 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:01.842216 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:01.842468 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:01.844468 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data/instance:
uuid: "a3659ab260614feebf1e56e739b3d506"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "9ab1dec2f5dc64fb047f47a152605a7d"
server_key_iv: "21fa1ac7c09cd867e6a80beab5319fa3"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.845034 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal/instance:
uuid: "a3659ab260614feebf1e56e739b3d506"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "9ab1dec2f5dc64fb047f47a152605a7d"
server_key_iv: "21fa1ac7c09cd867e6a80beab5319fa3"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.848774 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:36:01.851177 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:01.852473 13859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:01.852644 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "a3659ab260614feebf1e56e739b3d506"
format_stamp: "Formatted at 2026-09-27 16:36:01 on dist-test-slave-t27c"
server_key: "9ab1dec2f5dc64fb047f47a152605a7d"
server_key_iv: "21fa1ac7c09cd867e6a80beab5319fa3"
server_key_version: "encryptionkey@0"
I20260927 16:36:01.852777 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:01.892773 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:01.895713 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:01.895934 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:01.902779 13859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44907
I20260927 16:36:01.902796 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44907 every 8 connection(s)
I20260927 16:36:01.903714 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:36:01.906486 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:01.909199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 13859
I20260927 16:36:01.909355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526802550915-392-0/minicluster-data/master-2/wal/instance
I20260927 16:36:01.909605 392 external_mini_cluster.cc:1428] Setting key b09bf4e8dff64ed12e556d8b784a7057
I20260927 16:36:01.910218 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.912442 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 3
I20260927 16:36:01.916107 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } attempt: 3
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:36:01.925246 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.912812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43192)
0927 16:36:01.912922 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.912938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.913016 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.913508 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.913511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.913699 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.913794 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.913801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.915290 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.915293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.916090 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.916100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.917101 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.917105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.917203 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.917974 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.917990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.919475 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.923657 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.923661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.923663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.923931 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.924224 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.924234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.924236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.924337 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.924818 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.924824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.924926 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.924999 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.925102 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 16:36:01.925246 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.912173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.98.62:46113)
0927 16:36:01.912321 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.912326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.912658 (+ 332us) server_negotiation.cc:407] Connection header received
0927 16:36:01.912950 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.912954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.913003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.913059 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.914441 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.915175 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.916211 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.916535 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.916616 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.920380 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.920386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.920388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.920418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.922261 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.922954 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.922960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.922962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.923051 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.923471 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.923477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.923480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.923807 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.923923 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.924931 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.925052 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":768}
I20260927 16:36:01.926872 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.916501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43206)
0927 16:36:01.916749 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.916762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.916850 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.917722 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.917726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.917934 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.918038 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.918045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.919158 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.919161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.920043 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.920054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.920696 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.920701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.920800 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.921390 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.921409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.922851 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.925295 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.925298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.925300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.925539 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.925887 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.925893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.925895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.925987 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.926440 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.926443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.926532 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.926611 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.926733 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260927 16:36:01.926942 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.912079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46113 (local address 127.0.0.1:39476)
0927 16:36:01.912461 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.912508 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.912766 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.913310 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.913320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.913775 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.914306 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.914319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.915330 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.915332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.916029 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.916037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.916809 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.916813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.917035 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.917785 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.917803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.920212 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.922386 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.922397 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.922407 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.922827 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.923172 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.923175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.923178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.923361 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.924129 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.924134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.924687 (+ 553us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.925434 (+ 747us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.925782 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":71,"thread_start_us":139,"threads_started":1}
I20260927 16:36:01.927147 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.910934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.0.98.60:44907)
0927 16:36:01.911467 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.911482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.911517 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:36:01.912292 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.912315 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.912723 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.913842 (+ 1119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.914078 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.915078 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.916314 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.916935 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.917050 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.920711 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.920715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.920717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.920741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.923183 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.923720 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.923725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.923726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.923804 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.924237 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.924242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.924245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.924700 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.925484 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.925590 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.925970 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":341,"server-negotiator.queue_time_us":390,"spinlock_wait_cycles":768,"thread_start_us":197,"threads_started":1}
I20260927 16:36:01.926942 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.913047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.98.60:44907)
0927 16:36:01.913308 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.913314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.913326 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:36:01.913355 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.913357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.913389 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.913541 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.914081 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.915177 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.916240 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.916983 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.917416 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.919612 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.919621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.919635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.919662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.923556 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.924027 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.924032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.924034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.924124 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.924450 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.924454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.924457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.924694 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.926035 (+ 1341us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.926106 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.926248 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":415,"server-negotiator.queue_time_us":196,"spinlock_wait_cycles":896,"thread_start_us":116,"threads_started":1}
I20260927 16:36:01.928259 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:36:01.928369 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.916767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.0.98.60:44907)
0927 16:36:01.917463 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.917472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.917482 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:36:01.917508 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.917534 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.917580 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.917670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.918161 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.919033 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.920193 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.920551 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.921015 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.923022 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.923027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.923030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.923060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.925174 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.925655 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.925657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.925659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.925731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.926100 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.926103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.926104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.926321 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.927190 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.927283 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.928184 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":90,"threads_started":1}
I20260927 16:36:01.931788 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: Bootstrap starting.
I20260927 16:36:01.931788 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: Bootstrap starting.
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:36:01.934584 13851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:01.934584 13776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:01.935456 13776 log.cc:824] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:01.935456 13851 log.cc:824] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:01.937435 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.928801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.0.98.61:46239)
0927 16:36:01.928911 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.928928 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.929159 (+ 231us) server_negotiation.cc:407] Connection header received
0927 16:36:01.929284 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.929289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.929341 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.929392 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.929957 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.930615 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.931359 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.931618 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.931730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.933554 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.933558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.933560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.933584 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.935295 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.935831 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.935835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.935837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.935922 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.936347 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.936351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.936352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.936717 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.936814 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.937152 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.937272 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:36:01.937994 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: No bootstrap required, opened a new log
I20260927 16:36:01.938218 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.928717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46239 (local address 127.0.0.1:35688)
0927 16:36:01.929074 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.929088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.929167 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.929520 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.929523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.929698 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.929805 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.929814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.930725 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.930727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.931239 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.931245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.931730 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.931734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.931828 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.932307 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.932320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.933424 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.935432 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.935435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.935437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.935686 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.936044 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.936048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.936053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.936210 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.936859 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.936862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.936964 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.937829 (+ 865us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.937973 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260927 16:36:01.938431 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: No bootstrap required, opened a new log
I20260927 16:36:01.939853 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:01.942565 13776 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.942811 13776 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:01.942904 13776 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb3fc0ed1d442f192c04e46809cc95b, State: Initialized, Role: FOLLOWER
I20260927 16:36:01.943444 13776 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.944248 13851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.944471 13851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:01.944533 13851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eecce9fe2a0408d923d7e1a0e6e2dc7, State: Initialized, Role: FOLLOWER
I20260927 16:36:01.944873 13776 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:01.945016 13851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.945580 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.934311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.0.98.62:46113)
0927 16:36:01.934826 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.934830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.934846 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:36:01.934909 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.934913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.934956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.935038 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.936140 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.937000 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.938068 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.938439 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.938642 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.941468 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.941473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.941475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.941500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.943392 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.943900 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.943903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.943905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.943985 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.944268 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.944272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.944274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.944495 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.944666 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.945327 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.945442 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431}
I20260927 16:36:01.946339 13851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:36:01.951123 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.940370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44907 (local address 127.0.0.1:43218)
0927 16:36:01.940579 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:01.940592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:01.940660 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:01.940966 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:01.940971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:01.941138 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:01.941227 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.941234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.942876 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.942933 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.943870 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:01.943891 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.944657 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.944662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:01.944735 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.945299 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:01.945313 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:01.947569 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:01.949517 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.949520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.949522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.949747 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.950038 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:01.950041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:01.950042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:01.950128 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:01.950635 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:01.950637 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:01.950730 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 16:36:01.950826 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:36:01.950969 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 16:36:01.952193 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.940568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.0.98.60:44907)
0927 16:36:01.940691 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.940695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.940726 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:36:01.940762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.940764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.940804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.940879 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.941344 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.942658 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.944001 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.944326 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.944421 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.947712 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.947715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.947718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.947740 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.949407 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.949846 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.949849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.949851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.949944 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.950302 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.950305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.950307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.950507 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.950601 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.951921 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.952036 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 16:36:01 dist-test-slave-t27c krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 16:36:01.953606 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:36:01.953675 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:36:01.953855 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:01.953970 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:01.956111 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: Bootstrap starting.
I20260927 16:36:01.958027 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.947239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.61:46239)
0927 16:36:01.947596 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.947603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.947617 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:36:01.947653 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.947656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.947696 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.947784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.948685 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.949447 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.950499 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.950808 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.950901 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.953701 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.953705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.953707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.953745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.955762 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.956232 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.956235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.956238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.956329 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.956622 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.956626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.956627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.956878 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.957061 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.957754 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.957884 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260927 16:36:01.959178 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:01.960088 13927 log.cc:824] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: Log is configured to *not* fsync() on all Append() calls
W20260927 16:36:01.961257 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:36:01.961349 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:36:01.958516 13708 master_runner.cc:186] Error getting master registration for 127.0.98.60:44907: OK, instance_id { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" instance_seqno: 1790526961900827 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:36:01.961942 13708 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:36:01.962554 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: No bootstrap required, opened a new log
I20260927 16:36:01.964082 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:36:01.955523 13783 master_runner.cc:186] Error getting master registration for 127.0.98.60:44907: OK, instance_id { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" instance_seqno: 1790526961900827 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:36:01.966832 13783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:36:01.967509 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.967737 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:01.967803 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3659ab260614feebf1e56e739b3d506, State: Initialized, Role: FOLLOWER
I20260927 16:36:01.968282 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:01.968751 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:01.968924 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:01.969568 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:01.972504 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:01.972726 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:01.976979 13859 master_runner.cc:263] No leader master found from master a3659ab260614feebf1e56e739b3d506
I20260927 16:36:01.977044 13859 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:36:01.979790 13979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:36:01.979875 13979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:36:01.985261 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.970682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.0.98.62:46113)
0927 16:36:01.970812 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.970816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.972422 (+ 1606us) server_negotiation.cc:407] Connection header received
0927 16:36:01.972621 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.972624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.972677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.972732 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.973814 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.974569 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.975677 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.975994 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.976083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.981479 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.981483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.981486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.981514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.983333 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.983842 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.983846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.983848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.983949 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.984263 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.984267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.984268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.984487 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.984593 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.984976 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.985101 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:36:01.996021 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.972170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.0.98.60:44907)
0927 16:36:01.972277 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.972281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.986451 (+ 14170us) server_negotiation.cc:407] Connection header received
0927 16:36:01.986636 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.986639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.986692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.986743 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.987643 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.988421 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.989547 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.990067 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.990226 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:01.992044 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:01.992048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:01.992051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:01.992076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:01.993819 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.994332 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.994336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.994337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.994418 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:01.994707 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:01.994711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:01.994712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:01.994931 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:01.995061 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:01.995729 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 16:36:01.995849 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:36:02.004552 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:01.971511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.0.98.61:46239)
0927 16:36:01.971647 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:01.971654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:01.986385 (+ 14731us) server_negotiation.cc:407] Connection header received
0927 16:36:01.986575 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:01.986579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:01.986630 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:01.986693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:01.987532 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.988334 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.998459 (+ 10125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:01.998821 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:01.998914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:02.000580 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:02.000584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:02.000587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:02.000616 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:02.002328 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:02.003184 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:02.003188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:02.003190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:02.003287 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:02.003571 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:02.003574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:02.003576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:02.003788 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:02.003886 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:02.004218 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:36:02.004373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:36:02.017300 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:02.017500 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:02.018596 13967 leader_election.cc:288] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb3fc0ed1d442f192c04e46809cc95b (127.0.98.62:46113), 0eecce9fe2a0408d923d7e1a0e6e2dc7 (127.0.98.61:46239)
I20260927 16:36:02.019321 13750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3659ab260614feebf1e56e739b3d506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" is_pre_election: true
I20260927 16:36:02.019645 13750 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3659ab260614feebf1e56e739b3d506 in term 0.
I20260927 16:36:02.020078 13878 leader_election.cc:302] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3659ab260614feebf1e56e739b3d506, dbb3fc0ed1d442f192c04e46809cc95b; no voters:
I20260927 16:36:02.020649 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3659ab260614feebf1e56e739b3d506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" is_pre_election: true
I20260927 16:36:02.020916 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3659ab260614feebf1e56e739b3d506 in term 0.
I20260927 16:36:02.022970 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:36:02.023066 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:36:02.023126 13967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:02.024854 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:02.025462 13750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3659ab260614feebf1e56e739b3d506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb3fc0ed1d442f192c04e46809cc95b"
I20260927 16:36:02.025612 13750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:02.026942 13750 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3659ab260614feebf1e56e739b3d506 in term 1.
I20260927 16:36:02.027273 13878 leader_election.cc:302] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3659ab260614feebf1e56e739b3d506, dbb3fc0ed1d442f192c04e46809cc95b; no voters:
I20260927 16:36:02.029335 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3659ab260614feebf1e56e739b3d506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7"
I20260927 16:36:02.029462 13825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:02.030743 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3659ab260614feebf1e56e739b3d506 in term 1.
I20260927 16:36:02.031052 13967 leader_election.cc:288] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [CANDIDATE]: Term 1 election: Requested vote from peers dbb3fc0ed1d442f192c04e46809cc95b (127.0.98.62:46113), 0eecce9fe2a0408d923d7e1a0e6e2dc7 (127.0.98.61:46239)
I20260927 16:36:02.031175 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:02.031279 13967 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [term 1 LEADER]: Becoming Leader. State: Replica: a3659ab260614feebf1e56e739b3d506, State: Running, Role: LEADER
I20260927 16:36:02.031613 13967 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } }
I20260927 16:36:02.033478 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3659ab260614feebf1e56e739b3d506. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.033591 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:02.033963 13985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:02.036566 13985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:02.043854 13750 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [term 1 FOLLOWER]: Refusing update from remote peer a3659ab260614feebf1e56e739b3d506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:02.043859 13825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [term 1 FOLLOWER]: Refusing update from remote peer a3659ab260614feebf1e56e739b3d506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:02.044520 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:02.045845 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:02.049789 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3659ab260614feebf1e56e739b3d506. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.049902 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:02.050196 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3659ab260614feebf1e56e739b3d506. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.050303 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:02.052500 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.052632 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:02.053467 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.053577 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:02.054062 13986 mvcc.cc:205] Tried to move back new op lower bound from 7333998436523905024 to 7333998436484653056. Current Snapshot: MvccSnapshot[applied={T|T < 7333998436523905024}]
I20260927 16:36:02.058983 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.059091 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:36:02.058070 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3659ab260614feebf1e56e739b3d506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb3fc0ed1d442f192c04e46809cc95b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46113 } } peers { permanent_uuid: "0eecce9fe2a0408d923d7e1a0e6e2dc7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46239 } } peers { permanent_uuid: "a3659ab260614feebf1e56e739b3d506" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44907 } } }
I20260927 16:36:02.059342 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:02.059900 13985 catalog_manager.cc:1347] Generated new cluster ID: b63490417cb14aabaacc44b2fcdff07a
I20260927 16:36:02.059959 13985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:02.060297 13988 mvcc.cc:205] Tried to move back new op lower bound from 7333998436523905024 to 7333998436484653056. Current Snapshot: MvccSnapshot[applied={T|T < 7333998436523905024}]
I20260927 16:36:02.062047 13987 mvcc.cc:205] Tried to move back new op lower bound from 7333998436523905024 to 7333998436484653056. Current Snapshot: MvccSnapshot[applied={T|T < 7333998436523905024}]
I20260927 16:36:02.078150 13985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:36:02.079385 13985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:02.098397 13985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3659ab260614feebf1e56e739b3d506: Generated new TSK 0
I20260927 16:36:02.099325 13985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:36:02.954969 13959 catalog_manager.cc:1259] Loaded cluster ID: b63490417cb14aabaacc44b2fcdff07a
I20260927 16:36:02.955053 13959 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: loading cluster ID for follower catalog manager: success
I20260927 16:36:02.956740 13959 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: acquiring CA information for follower catalog manager: success
I20260927 16:36:02.957903 13959 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0eecce9fe2a0408d923d7e1a0e6e2dc7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:36:02.962484 13948 catalog_manager.cc:1259] Loaded cluster ID: b63490417cb14aabaacc44b2fcdff07a
I20260927 16:36:02.962544 13948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: loading cluster ID for follower catalog manager: success
I20260927 16:36:02.964160 13948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: acquiring CA information for follower catalog manager: success
I20260927 16:36:02.965293 13948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dbb3fc0ed1d442f192c04e46809cc95b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:36:03.170396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13708
I20260927 16:36:03.177321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13783
I20260927 16:36:03.183282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 13859
2026-09-27T16:36:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3095 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:36:03 dist-test-slave-t27c krb5kdc[14000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:36:03 dist-test-slave-t27c krb5kdc[14000](info): set up 2 sockets
Sep 27 16:36:03 dist-test-slave-t27c krb5kdc[14000](info): commencing operation
krb5kdc: starting...
W20260927 16:36:04.230496 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:36:04.349085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:04.458182 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:04.458444 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:04.458494 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:04.462409 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:04.462459 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:04.462478 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:04.462494 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:04.462508 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:04.467774 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:04.468814 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:04.469801 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:04.478278 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:04.478368 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:04.478278 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:04.478381 14013 server_base.cc:1034] running on GCE node
I20260927 16:36:04.479089 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:04.480296 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:04.481484 14013 hybrid_clock.cc:648] HybridClock initialized: now 1790526964481466 us; error 28 us; skew 500 ppm
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:04.484163 14013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:04.485006 14013 webserver.cc:460] Webserver started at http://127.0.98.62:45313/ using document root <none> and password file <none>
I20260927 16:36:04.485553 14013 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:04.485641 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:04.485869 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:04.487915 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "f7553c778ac94c9db88d887775a64521"
format_stamp: "Formatted at 2026-09-27 16:36:04 on dist-test-slave-t27c"
server_key: "1adf4229c20c9d3c5c25ebe7a04ee4d2"
server_key_iv: "0aed6d93ce67e09ae1fb07db0437cc0c"
server_key_version: "encryptionkey@0"
I20260927 16:36:04.488477 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7553c778ac94c9db88d887775a64521"
format_stamp: "Formatted at 2026-09-27 16:36:04 on dist-test-slave-t27c"
server_key: "1adf4229c20c9d3c5c25ebe7a04ee4d2"
server_key_iv: "0aed6d93ce67e09ae1fb07db0437cc0c"
server_key_version: "encryptionkey@0"
I20260927 16:36:04.492143 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:36:04.494508 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:04.495826 14013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:36:04.495967 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "f7553c778ac94c9db88d887775a64521"
format_stamp: "Formatted at 2026-09-27 16:36:04 on dist-test-slave-t27c"
server_key: "1adf4229c20c9d3c5c25ebe7a04ee4d2"
server_key_iv: "0aed6d93ce67e09ae1fb07db0437cc0c"
server_key_version: "encryptionkey@0"
I20260927 16:36:04.496096 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:04.521163 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:04.524091 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:04.524298 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:04.530989 14013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34963
I20260927 16:36:04.531002 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34963 every 8 connection(s)
I20260927 16:36:04.531913 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:04.534689 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:04.535254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14013
I20260927 16:36:04.535380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:36:04.535683 392 external_mini_cluster.cc:1428] Setting key 30f56803e826b716760fc1cd8a64cef8
I20260927 16:36:04.542132 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521: Bootstrap starting.
I20260927 16:36:04.544929 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521: Neither blocks nor log segments found. Creating new log.
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:04.545706 14081 log.cc:824] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:04.547832 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521: No bootstrap required, opened a new log
I20260927 16:36:04.551159 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:04.537000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.0.98.62:34963)
0927 16:36:04.537468 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:04.537490 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:04.537528 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:36:04.538255 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:04.538278 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:04.538745 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:04.539153 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:04.540080 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:04.541057 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:04.542279 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:04.542751 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:04.542865 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:04.546016 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:04.546023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:04.546054 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:04.546089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:04.548307 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:04.548764 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:04.548770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:04.548775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:04.548894 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:04.549184 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:04.549188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:04.549189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:04.549573 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:04.549791 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:04.550071 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 16:36:04.550414 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260927 16:36:04.552762 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260927 16:36:04.553043 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:04.553130 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7553c778ac94c9db88d887775a64521, State: Initialized, Role: FOLLOWER
I20260927 16:36:04.553606 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260927 16:36:04.553776 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:04.553845 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:04.553941 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:04.553979 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:36:04.554889 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260927 16:36:04.555258 14081 leader_election.cc:302] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7553c778ac94c9db88d887775a64521; no voters:
I20260927 16:36:04.555595 14081 leader_election.cc:288] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:36:04.555718 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:04.555938 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [term 1 LEADER]: Becoming Leader. State: Replica: f7553c778ac94c9db88d887775a64521, State: Running, Role: LEADER
I20260927 16:36:04.556275 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } }
I20260927 16:36:04.556816 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:04.557847 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7553c778ac94c9db88d887775a64521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } } }
I20260927 16:36:04.557968 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:04.558353 14095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:04.558344 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7553c778ac94c9db88d887775a64521. Latest consensus state: current_term: 1 leader_uuid: "f7553c778ac94c9db88d887775a64521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7553c778ac94c9db88d887775a64521" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34963 } } }
I20260927 16:36:04.559051 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:04.561414 14095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:04.566670 14095 catalog_manager.cc:1347] Generated new cluster ID: 411421fc56194d14b78c86f236c66a19
I20260927 16:36:04.566730 14095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:04.589046 14095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:36:04.590404 14095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:04.600389 14095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7553c778ac94c9db88d887775a64521: Generated new TSK 0
I20260927 16:36:04.601249 14095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:36:04.609535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14013
2026-09-27T16:36:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14108](info): set up 2 sockets
Sep 27 16:36:04 dist-test-slave-t27c krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260927 16:36:05.654461 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:36:05 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:36:05.773525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:05.888861 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:05.889123 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:05.889163 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:05.893378 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:05.893448 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:05.893469 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:05.893486 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:05.893500 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:05.898808 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:05.899870 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:05.900942 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:05.910120 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:05.910112 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:05.910112 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:05.910148 14121 server_base.cc:1034] running on GCE node
I20260927 16:36:05.910900 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:05.911973 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:05.913134 14121 hybrid_clock.cc:648] HybridClock initialized: now 1790526965913119 us; error 26 us; skew 500 ppm
Sep 27 16:36:05 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:05.920657 14121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:05.921609 14121 webserver.cc:460] Webserver started at http://127.0.98.62:45663/ using document root <none> and password file <none>
I20260927 16:36:05.922252 14121 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:05.922318 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:05.922550 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:05.924594 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "db6fb4e2651640e4806229b12d185d24"
format_stamp: "Formatted at 2026-09-27 16:36:05 on dist-test-slave-t27c"
server_key: "9ccd52adc60c43d85b20de88aa56b918"
server_key_iv: "eba9247882f26a2f0bfc7719c669feaf"
server_key_version: "encryptionkey@0"
I20260927 16:36:05.925179 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "db6fb4e2651640e4806229b12d185d24"
format_stamp: "Formatted at 2026-09-27 16:36:05 on dist-test-slave-t27c"
server_key: "9ccd52adc60c43d85b20de88aa56b918"
server_key_iv: "eba9247882f26a2f0bfc7719c669feaf"
server_key_version: "encryptionkey@0"
I20260927 16:36:05.928910 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:36:05.931237 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:05.932423 14121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:36:05.932565 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "db6fb4e2651640e4806229b12d185d24"
format_stamp: "Formatted at 2026-09-27 16:36:05 on dist-test-slave-t27c"
server_key: "9ccd52adc60c43d85b20de88aa56b918"
server_key_iv: "eba9247882f26a2f0bfc7719c669feaf"
server_key_version: "encryptionkey@0"
I20260927 16:36:05.932689 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:05.948290 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:05.951870 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:05.952095 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:05.959095 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43815
I20260927 16:36:05.959386 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43815 every 8 connection(s)
I20260927 16:36:05.960088 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:05.962814 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:05.968376 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap starting.
I20260927 16:36:05.970101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14121
I20260927 16:36:05.970266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:36:05.970549 392 external_mini_cluster.cc:1428] Setting key b6e77887ec2669f2710af4a2807c9332
I20260927 16:36:05.971073 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:05.971949 14189 log.cc:824] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:05.973852 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: No bootstrap required, opened a new log
I20260927 16:36:05.977280 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:05.977520 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:05.977578 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Initialized, Role: FOLLOWER
I20260927 16:36:05.978073 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:05.978251 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:05.978341 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:05.978441 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:36:05 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:05.979686 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:05.980069 14189 leader_election.cc:302] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6fb4e2651640e4806229b12d185d24; no voters:
I20260927 16:36:05.980377 14189 leader_election.cc:288] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:36:05.980738 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:05.981261 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 LEADER]: Becoming Leader. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Running, Role: LEADER
I20260927 16:36:05.981640 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:05.981655 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:05.983492 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:05.983655 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:05.984305 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6fb4e2651640e4806229b12d185d24. Latest consensus state: current_term: 1 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:05.984467 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:05.986346 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:05.971683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.98.62:43815)
0927 16:36:05.972164 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:05.972183 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:05.972219 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:36:05.973031 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:05.973051 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:05.973384 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:05.973733 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:05.974601 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:05.975572 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:05.976428 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:05.976906 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:05.977078 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:05.979985 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:05.979991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:05.980004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:05.980031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:05.982326 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:05.982886 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:05.982893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:05.982899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:05.983046 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:05.983390 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:05.983394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:05.983395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:05.983949 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:05.984168 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:05.984968 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 16:36:05.985299 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
W20260927 16:36:05.986889 14209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:36:05.986976 14209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:36:05.987068 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:05.988116 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:05.993382 14208 catalog_manager.cc:1347] Generated new cluster ID: 3d9c8e1f7a9348d48f5c8c732159c2f7
I20260927 16:36:05.993443 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:06.005739 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:36:06.006960 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:06.012545 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Generated new TSK 0
I20260927 16:36:06.013221 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:36:06.045061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:06.158325 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:06.158588 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:06.158630 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:06.162725 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:06.162781 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:06.162839 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:36:06.167958 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:06.169269 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:06.170538 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:06.182062 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.182062 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.182085 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:06.183387 14215 server_base.cc:1034] running on GCE node
I20260927 16:36:06.183849 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:06.184643 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:06.185812 14215 hybrid_clock.cc:648] HybridClock initialized: now 1790526966185779 us; error 48 us; skew 500 ppm
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:06.188782 14215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:36:06.189740 14215 webserver.cc:460] Webserver started at http://127.0.98.1:33285/ using document root <none> and password file <none>
I20260927 16:36:06.190421 14215 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:06.190515 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:06.190812 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:06.192931 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "39a4cf3c60924b4f9990ed57bed7be57"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "774270dc2c957da47e482214aaca1414"
server_key_iv: "b5978010df28f0baa3b196ad7bcb8854"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.193511 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39a4cf3c60924b4f9990ed57bed7be57"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "774270dc2c957da47e482214aaca1414"
server_key_iv: "b5978010df28f0baa3b196ad7bcb8854"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.197266 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:36:06.199786 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.201059 14215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:36:06.201191 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "39a4cf3c60924b4f9990ed57bed7be57"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "774270dc2c957da47e482214aaca1414"
server_key_iv: "b5978010df28f0baa3b196ad7bcb8854"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.201303 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:06.212163 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:06.215487 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:06.215698 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:06.216277 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:06.217082 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:06.217155 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.217226 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:06.217278 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.226549 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37005
I20260927 16:36:06.226617 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37005 every 8 connection(s)
I20260927 16:36:06.227582 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:36:06.231562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14215
I20260927 16:36:06.231706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:36:06.232015 392 external_mini_cluster.cc:1428] Setting key 5d685af606bf578e5462083e80e03e3e
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:36:06.243245 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.229565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38021 (local address 127.0.98.62:43815)
0927 16:36:06.229703 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.229708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.230857 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 16:36:06.232172 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.232175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.232228 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.232305 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:06.233990 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.234604 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.236188 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.236660 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.236804 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.239393 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:06.239400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:06.239405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:06.239441 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:06.241196 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.241694 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.241701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.241706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.241799 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.242129 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.242133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.242135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.242400 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:06.242507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.242970 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.243116 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:36:06.244203 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.229937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:38021)
0927 16:36:06.230658 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.230722 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.231866 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.232545 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.232553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.233026 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:06.233823 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.233845 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.234735 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.234741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.236054 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.236064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.236873 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.236877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.237022 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.237666 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:06.237682 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:06.239218 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:06.241316 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.241321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.241332 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.241577 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.241898 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.241900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.241902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.242029 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.242522 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:06.242527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:06.242760 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.242918 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.243264 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:36:06.245464 14344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 16:36:06.245723 14344 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:06.246366 14344 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:06.247998 14153 ts_manager.cc:194] Registered new tserver with Master: 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:06.249120 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:36:06.260536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:06.371932 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:06.372228 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:06.372323 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:06.376487 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:06.376555 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:06.376633 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:36:06.381877 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:06.383108 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:06.384317 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:06.394420 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.394490 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.394429 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:06.394686 14349 server_base.cc:1034] running on GCE node
I20260927 16:36:06.395144 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:06.395787 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:06.396998 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790526966396981 us; error 29 us; skew 500 ppm
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:06.399748 14349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:36:06.400630 14349 webserver.cc:460] Webserver started at http://127.0.98.2:43881/ using document root <none> and password file <none>
I20260927 16:36:06.401193 14349 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:06.401283 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:06.401515 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:06.403611 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "21479876f0e74df984203cd6e36a4f28"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "7e8356b2e30cf47bcd5fbb2c3f6c9a59"
server_key_iv: "5d202432e59db1c06870600af0698a61"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.404251 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21479876f0e74df984203cd6e36a4f28"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "7e8356b2e30cf47bcd5fbb2c3f6c9a59"
server_key_iv: "5d202432e59db1c06870600af0698a61"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.408075 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:36:06.410435 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.411615 14349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:06.411746 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "21479876f0e74df984203cd6e36a4f28"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "7e8356b2e30cf47bcd5fbb2c3f6c9a59"
server_key_iv: "5d202432e59db1c06870600af0698a61"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.411854 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:06.427424 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:06.430542 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:06.430775 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:06.431387 14349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:06.432240 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:06.432312 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.432386 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:06.432443 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.441383 14349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37137
I20260927 16:36:06.441403 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37137 every 8 connection(s)
I20260927 16:36:06.442463 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:36:06.446738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14349
I20260927 16:36:06.446884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:36:06.447163 392 external_mini_cluster.cc:1428] Setting key 54a97c98c926de51e77591061546b073
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:36:06.457521 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.444269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35955 (local address 127.0.98.62:43815)
0927 16:36:06.444404 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.444407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.445385 (+ 978us) server_negotiation.cc:407] Connection header received
0927 16:36:06.446321 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.446324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.446383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.446489 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:06.447889 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.448486 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.450293 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.450682 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.450804 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.453605 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:06.453608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:06.453610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:06.453634 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:06.455202 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.455832 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.455835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.455836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.455908 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.456305 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.456307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.456308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.456472 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:06.456547 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.457216 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.457317 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:36:06.458531 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.444599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:35955)
0927 16:36:06.445235 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.445275 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.446068 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.446668 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.446675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.447117 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:06.447746 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.447764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.448635 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.448639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.450105 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.450121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.450829 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.450832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.450975 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.451601 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:06.451617 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:06.453430 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:06.455326 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.455332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.455348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.455732 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.456001 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.456006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.456010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.456207 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.456585 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:06.456593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:06.456954 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.457185 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.457558 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":114,"threads_started":1}
I20260927 16:36:06.459740 14478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:36:06.460005 14478 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:06.460633 14478 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:06.461894 14153 ts_manager.cc:194] Registered new tserver with Master: 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
I20260927 16:36:06.462781 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:36:06.475265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:06.601505 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:06.601900 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:06.601998 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:06.607013 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:06.607077 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:06.607136 14483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:36:06.612293 14483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:06.613269 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:06.614291 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:06.624138 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.624171 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:06.624194 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:06.624539 14483 server_base.cc:1034] running on GCE node
I20260927 16:36:06.625105 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:06.625903 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:06.627112 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790526966627082 us; error 67 us; skew 500 ppm
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:06.630309 14483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:36:06.631246 14483 webserver.cc:460] Webserver started at http://127.0.98.3:45659/ using document root <none> and password file <none>
I20260927 16:36:06.631856 14483 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:06.631958 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:06.632182 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:06.634305 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "e5e426b8769c4d0eb1e71f9ba682d66c"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "dd72343c453ca7dcb130b73cf893064b"
server_key_iv: "6134c61875ca193522eabe727a4b3b69"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.634873 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e5e426b8769c4d0eb1e71f9ba682d66c"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "dd72343c453ca7dcb130b73cf893064b"
server_key_iv: "6134c61875ca193522eabe727a4b3b69"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.638875 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:36:06.641117 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.642374 14483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 16:36:06.642484 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "e5e426b8769c4d0eb1e71f9ba682d66c"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "dd72343c453ca7dcb130b73cf893064b"
server_key_iv: "6134c61875ca193522eabe727a4b3b69"
server_key_version: "encryptionkey@0"
I20260927 16:36:06.642578 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:06.658066 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:06.661362 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:06.661515 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:06.662109 14483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:06.662937 14483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:06.662983 14483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.663021 14483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:06.663043 14483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:06.671861 14483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43757
I20260927 16:36:06.671881 14611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43757 every 8 connection(s)
I20260927 16:36:06.672793 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:36:06.682080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14483
Sep 27 16:36:06 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:06.682236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:36:06.682578 392 external_mini_cluster.cc:1428] Setting key f7581e166f168df69b1a9d16d2b92c61
I20260927 16:36:06.686754 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.674676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47663 (local address 127.0.98.62:43815)
0927 16:36:06.674824 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.674828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.675853 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 16:36:06.676756 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.676759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.676818 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.676929 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:06.678142 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.678759 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.679929 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.680157 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.680266 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.683043 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:06.683049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:06.683051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:06.683080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:06.684881 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.685339 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.685341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.685342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.685419 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.685724 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.685726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.685727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.685887 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:06.685961 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.686536 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.686634 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:36:06.687597 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.675041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:47663)
0927 16:36:06.675690 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.675732 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.676544 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.677107 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.677114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.677479 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:06.677985 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.677998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.678893 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.678899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.679787 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.679796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.680316 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.680320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.680492 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.681057 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:06.681073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:06.682835 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:06.684985 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.684990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.685001 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.685252 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.685501 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:06.685503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:06.685504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:06.685645 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:06.685971 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:06.685976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:06.686297 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.686483 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.686749 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":110,"threads_started":1}
I20260927 16:36:06.688671 14612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 16:36:06.688913 14612 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:06.689388 14612 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:06.690405 14153 ts_manager.cc:194] Registered new tserver with Master: e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757)
I20260927 16:36:06.691043 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47663
I20260927 16:36:06.696636 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:36:06.705962 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.698579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.98.62:43815)
0927 16:36:06.698727 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.698731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.698919 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:36:06.699003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.699005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.699042 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.699108 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:06.699854 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.700364 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.701123 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.701336 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.701409 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.702755 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:06.702758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:06.702761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:06.702784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:06.704048 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.704471 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.704474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.704476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.704572 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:06.704828 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:06.704830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:06.704832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:06.705130 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:06.705252 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.705538 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.705688 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:36:06.709270 14153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:36:06.712713 14153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:36:06.726974 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.721664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43757 (local address 127.0.0.1:48346)
0927 16:36:06.722295 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.722307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.722389 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.722844 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.722850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.722872 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:06.723033 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.723039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724163 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.724170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.725331 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.725340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.726435 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.726439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.726549 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.726591 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.726703 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.726814 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":7,"thread_start_us":80,"threads_started":1}
I20260927 16:36:06.727020 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.722031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.3:43757)
0927 16:36:06.722374 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.722385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.722414 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:36:06.722474 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.722477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.722689 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.722844 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:06.723232 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724028 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.725456 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.726318 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.726423 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.726563 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.726777 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.726873 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":145,"threads_started":1}
I20260927 16:36:06.727573 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.720935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37137 (local address 127.0.0.1:51394)
0927 16:36:06.721815 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.721866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.721997 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.723025 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.723028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.723044 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:06.723152 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.723157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724636 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.724640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.725512 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.725522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.727209 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.727213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.727299 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.727320 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.727382 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.727448 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":74,"threads_started":1}
I20260927 16:36:06.728021 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.721138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.0.98.2:37137)
0927 16:36:06.722611 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.722623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.722643 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:36:06.722710 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.722715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.722876 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.723038 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:06.723272 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724472 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.726300 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.727108 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.727213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.727352 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.727405 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.727872 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1383,"thread_start_us":155,"threads_started":1}
I20260927 16:36:06.728375 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.721377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.0.1:34592)
0927 16:36:06.721940 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.721955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.722036 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.722861 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.722865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.722881 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:06.723235 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.723243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724636 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.724640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.726580 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.726592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.727922 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.727925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.728011 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.728033 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.728175 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.728253 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1053,"thread_start_us":102,"threads_started":1}
I20260927 16:36:06.728933 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.721560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.98.1:37005)
0927 16:36:06.722341 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.722357 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.722380 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:36:06.722462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.722468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.722705 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.722983 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:06.723362 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.724493 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.726714 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.727787 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.728499 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.728654 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.728710 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.728815 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":88,"threads_started":1}
I20260927 16:36:06.730525 14546 tablet_service.cc:1467] Processing CreateTablet for tablet f5a8fee875654450ad1dfe8df92cece6 (DEFAULT_TABLE table=test-table [id=e4ce627b1fb34f019f61b72df6595c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:06.730808 14412 tablet_service.cc:1467] Processing CreateTablet for tablet f5a8fee875654450ad1dfe8df92cece6 (DEFAULT_TABLE table=test-table [id=e4ce627b1fb34f019f61b72df6595c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:06.731122 14278 tablet_service.cc:1467] Processing CreateTablet for tablet f5a8fee875654450ad1dfe8df92cece6 (DEFAULT_TABLE table=test-table [id=e4ce627b1fb34f019f61b72df6595c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:06.731693 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a8fee875654450ad1dfe8df92cece6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:06.731693 14546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a8fee875654450ad1dfe8df92cece6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:06.732133 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a8fee875654450ad1dfe8df92cece6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:06.738250 14629 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap starting.
I20260927 16:36:06.739938 14631 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Bootstrap starting.
I20260927 16:36:06.740731 14629 tablet_bootstrap.cc:654] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:06.741781 14629 log.cc:824] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:06.742091 14631 tablet_bootstrap.cc:654] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:06.742527 14630 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Bootstrap starting.
I20260927 16:36:06.742957 14631 log.cc:824] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:06.743970 14629 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: No bootstrap required, opened a new log
I20260927 16:36:06.744175 14629 ts_tablet_manager.cc:1397] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:36:06.744830 14630 tablet_bootstrap.cc:654] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:06.745076 14631 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: No bootstrap required, opened a new log
I20260927 16:36:06.745265 14631 ts_tablet_manager.cc:1397] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:36:06.746119 14630 log.cc:824] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:06.747897 14630 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: No bootstrap required, opened a new log
I20260927 16:36:06.748111 14630 ts_tablet_manager.cc:1397] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:36:06.748451 14629 raft_consensus.cc:348] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.748739 14629 raft_consensus.cc:374] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:06.748800 14629 raft_consensus.cc:729] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a4cf3c60924b4f9990ed57bed7be57, State: Initialized, Role: FOLLOWER
I20260927 16:36:06.749147 14631 raft_consensus.cc:348] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.749264 14629 consensus_queue.cc:260] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.749408 14631 raft_consensus.cc:374] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:06.749475 14631 raft_consensus.cc:729] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21479876f0e74df984203cd6e36a4f28, State: Initialized, Role: FOLLOWER
I20260927 16:36:06.749933 14631 consensus_queue.cc:260] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.750326 14629 ts_tablet_manager.cc:1428] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 16:36:06.750682 14344 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 16:36:06.750682 14478 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 16:36:06.750959 14631 ts_tablet_manager.cc:1428] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 16:36:06.752708 14630 raft_consensus.cc:348] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.752909 14630 raft_consensus.cc:374] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:06.752940 14630 raft_consensus.cc:729] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e426b8769c4d0eb1e71f9ba682d66c, State: Initialized, Role: FOLLOWER
I20260927 16:36:06.753330 14630 consensus_queue.cc:260] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.753978 14612 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 16:36:06.754307 14630 ts_tablet_manager.cc:1428] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:36:06.891937 14637 raft_consensus.cc:482] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:06.892130 14637 raft_consensus.cc:504] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.893270 14637 leader_election.cc:288] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137), 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:06.898121 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.893709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54225 (local address 127.0.98.2:37137)
0927 16:36:06.893847 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.893853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.894333 (+ 480us) server_negotiation.cc:407] Connection header received
0927 16:36:06.894474 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.894477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.894530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.894607 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:06.894836 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.895495 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.896828 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.897471 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.897581 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.897648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.897913 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.897985 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:36:06.898129 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.893683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59939 (local address 127.0.98.1:37005)
0927 16:36:06.893811 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.893816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.893834 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:36:06.893951 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.893957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.894028 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.894119 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:06.894476 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.895278 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.896828 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.897471 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.897581 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.897648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.897913 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.897986 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:36:06.898201 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.893575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.98.3:59939)
0927 16:36:06.893709 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.893727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.893844 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.894144 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.894150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.894214 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:06.894353 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.894360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.895385 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.895387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.896665 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.896676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.897599 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.897603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.897758 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.897802 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.897885 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.898000 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":36}
I20260927 16:36:06.898625 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.893575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37137 (local address 127.0.98.3:54225)
0927 16:36:06.894239 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:06.894252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:06.894344 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:06.894626 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:06.894628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:06.894642 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:06.894731 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.894735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.895594 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.895597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.896724 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:06.896733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.897599 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.897603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:06.897761 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.897807 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 16:36:06.897885 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:36:06.897999 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":368,"thread_start_us":145,"threads_started":1}
I20260927 16:36:06.898972 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a8fee875654450ad1dfe8df92cece6" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a4cf3c60924b4f9990ed57bed7be57" is_pre_election: true
I20260927 16:36:06.899330 14298 raft_consensus.cc:2454] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 0.
I20260927 16:36:06.899372 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a8fee875654450ad1dfe8df92cece6" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21479876f0e74df984203cd6e36a4f28" is_pre_election: true
I20260927 16:36:06.899652 14432 raft_consensus.cc:2454] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 0.
I20260927 16:36:06.899829 14500 leader_election.cc:302] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57, e5e426b8769c4d0eb1e71f9ba682d66c; no voters:
I20260927 16:36:06.900113 14637 raft_consensus.cc:2781] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:36:06.900200 14637 raft_consensus.cc:482] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:36:06.900245 14637 raft_consensus.cc:3037] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:06.901271 14637 raft_consensus.cc:504] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.901604 14637 leader_election.cc:288] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 election: Requested vote from peers 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137), 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:06.901938 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a8fee875654450ad1dfe8df92cece6" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21479876f0e74df984203cd6e36a4f28"
I20260927 16:36:06.901973 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a8fee875654450ad1dfe8df92cece6" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a4cf3c60924b4f9990ed57bed7be57"
I20260927 16:36:06.902076 14432 raft_consensus.cc:3037] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:06.902097 14298 raft_consensus.cc:3037] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:06.903242 14298 raft_consensus.cc:2454] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 1.
I20260927 16:36:06.903333 14432 raft_consensus.cc:2454] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 1.
I20260927 16:36:06.903570 14500 leader_election.cc:302] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57, e5e426b8769c4d0eb1e71f9ba682d66c; no voters:
I20260927 16:36:06.903759 14637 raft_consensus.cc:2781] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:06.904018 14637 raft_consensus.cc:686] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 LEADER]: Becoming Leader. State: Replica: e5e426b8769c4d0eb1e71f9ba682d66c, State: Running, Role: LEADER
I20260927 16:36:06.904402 14637 consensus_queue.cc:237] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:06.907963 14151 catalog_manager.cc:5495] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } health_report { overall_health: UNKNOWN } } }
W20260927 16:36:06.923727 14613 tablet.cc:2369] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:36:06.943652 14479 tablet.cc:2369] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:36:06.964797 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:06.960708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.0.98.3:43757)
0927 16:36:06.960862 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:06.960867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:06.960884 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:36:06.960994 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:06.960997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:06.961069 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:06.961191 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:06.961392 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.962197 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.963153 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:06.963500 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:06.963601 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:06.963879 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:06.964415 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:36:06.964539 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:06.964582 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 16:36:06.964638 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:36:06.970943 14298 raft_consensus.cc:1261] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Refusing update from remote peer e5e426b8769c4d0eb1e71f9ba682d66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:06.970969 14432 raft_consensus.cc:1261] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Refusing update from remote peer e5e426b8769c4d0eb1e71f9ba682d66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:06.971681 14637 consensus_queue.cc:1035] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:06.971946 14639 consensus_queue.cc:1035] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:36:06.978986 14345 tablet.cc:2369] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:36:06.980422 14642 mvcc.cc:205] Tried to move back new op lower bound from 7333998456705581056 to 7333998456443310080. Current Snapshot: MvccSnapshot[applied={T|T < 7333998456705581056}]
I20260927 16:36:06.981104 14644 mvcc.cc:205] Tried to move back new op lower bound from 7333998456705581056 to 7333998456443310080. Current Snapshot: MvccSnapshot[applied={T|T < 7333998456705581056}]
I20260927 16:36:06.982120 14643 mvcc.cc:205] Tried to move back new op lower bound from 7333998456705581056 to 7333998456443310080. Current Snapshot: MvccSnapshot[applied={T|T < 7333998456705581056}]
I20260927 16:36:06.989667 14151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:06.989864 14151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:06.992376 14151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Sending DeleteTablet for 3 replicas of tablet f5a8fee875654450ad1dfe8df92cece6
I20260927 16:36:06.993212 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a8fee875654450ad1dfe8df92cece6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48346
I20260927 16:36:06.993235 14412 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a8fee875654450ad1dfe8df92cece6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51394
I20260927 16:36:06.993227 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a8fee875654450ad1dfe8df92cece6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260927 16:36:06.993765 14652 tablet_replica.cc:330] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: stopping tablet replica
I20260927 16:36:06.993778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14215
I20260927 16:36:06.994117 14652 raft_consensus.cc:2229] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:06.994450 14652 raft_consensus.cc:2258] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:06.994905 14654 tablet_replica.cc:330] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: stopping tablet replica
I20260927 16:36:06.995219 14654 raft_consensus.cc:2229] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:36:06.995625 14654 raft_consensus.cc:2258] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:06.997076 14652 ts_tablet_manager.cc:1905] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:06.997084 14654 ts_tablet_manager.cc:1905] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:06.999468 14652 ts_tablet_manager.cc:1918] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:06.999588 14652 log.cc:1196] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/f5a8fee875654450ad1dfe8df92cece6
I20260927 16:36:06.999881 14652 ts_tablet_manager.cc:1939] T f5a8fee875654450ad1dfe8df92cece6 P 21479876f0e74df984203cd6e36a4f28: Deleting consensus metadata
I20260927 16:36:07.000670 14654 ts_tablet_manager.cc:1918] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:07.000772 14654 log.cc:1196] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/f5a8fee875654450ad1dfe8df92cece6
I20260927 16:36:07.001166 14654 ts_tablet_manager.cc:1939] T f5a8fee875654450ad1dfe8df92cece6 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting consensus metadata
I20260927 16:36:07.001271 14137 catalog_manager.cc:4841] TS 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): tablet f5a8fee875654450ad1dfe8df92cece6 (table test-table [id=e4ce627b1fb34f019f61b72df6595c98]) successfully deleted
W20260927 16:36:07.001751 14138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37005 (error 108)
I20260927 16:36:07.002593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14349
I20260927 16:36:07.002614 14137 catalog_manager.cc:4841] TS e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): tablet f5a8fee875654450ad1dfe8df92cece6 (table test-table [id=e4ce627b1fb34f019f61b72df6595c98]) successfully deleted
I20260927 16:36:07.002794 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.002519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.0.1:34596)
0927 16:36:07.002631 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.002700 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 16:36:07.003213 14138 catalog_manager.cc:4584] TS 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5a8fee875654450ad1dfe8df92cece6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37005: connect: Connection refused (error 111)
I20260927 16:36:07.008695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14483
I20260927 16:36:07.014649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14121
I20260927 16:36:07.021088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--webserver_interface=127.0.98.62
--webserver_port=45663
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:07.131644 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:07.131946 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:07.132022 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:07.136004 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:07.136104 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:07.136153 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:07.136214 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:07.136252 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:07.141448 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:07.142611 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:07.143694 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:07.157124 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.157132 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.157138 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:07.157500 14656 server_base.cc:1034] running on GCE node
I20260927 16:36:07.158075 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:07.159343 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:07.160532 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790526967160493 us; error 51 us; skew 500 ppm
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:07.163439 14656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:07.164372 14656 webserver.cc:460] Webserver started at http://127.0.98.62:45663/ using document root <none> and password file <none>
I20260927 16:36:07.164966 14656 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:07.165073 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:07.168924 14656 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:36:07.171072 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.172258 14656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:07.172401 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "db6fb4e2651640e4806229b12d185d24"
format_stamp: "Formatted at 2026-09-27 16:36:05 on dist-test-slave-t27c"
server_key: "9ccd52adc60c43d85b20de88aa56b918"
server_key_iv: "eba9247882f26a2f0bfc7719c669feaf"
server_key_version: "encryptionkey@0"
I20260927 16:36:07.173048 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:07.186369 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:07.189987 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:07.190297 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:07.198390 14656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43815
I20260927 16:36:07.198683 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43815 every 8 connection(s)
I20260927 16:36:07.199625 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:07.204106 14725 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:36:07.205716 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap starting.
I20260927 16:36:07.207681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14656
I20260927 16:36:07.208110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37005
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=33285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:07.223137 14725 log.cc:824] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:07.232621 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:07.233043 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap complete.
I20260927 16:36:07.237124 14725 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:07.237519 14725 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Initialized, Role: FOLLOWER
I20260927 16:36:07.238034 14725 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:07.238148 14725 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:07.238270 14725 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:07.238387 14725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:36:07.240268 14725 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:07.240602 14725 leader_election.cc:302] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6fb4e2651640e4806229b12d185d24; no voters:
I20260927 16:36:07.240965 14725 leader_election.cc:288] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:36:07.241055 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:36:07.241267 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 LEADER]: Becoming Leader. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Running, Role: LEADER
I20260927 16:36:07.241564 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:07.242215 14725 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:07.242599 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6fb4e2651640e4806229b12d185d24. Latest consensus state: current_term: 2 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:07.242726 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:07.242955 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:07.243062 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:07.243167 14733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:07.247936 14733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:07.248734 14733 catalog_manager.cc:1259] Loaded cluster ID: 3d9c8e1f7a9348d48f5c8c732159c2f7
I20260927 16:36:07.248798 14733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:07.250226 14733 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 16:36:07.255349 14733 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 16:36:07.260864 14733 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:07.261662 14733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Loaded TSK: 0
I20260927 16:36:07.262923 14733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:36:07.324438 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:07.324735 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:07.324812 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:07.328765 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:07.328861 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:07.328954 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:36:07.334275 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33285
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:07.335299 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:07.336385 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:07.346738 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.346738 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.346822 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:07.347869 14727 server_base.cc:1034] running on GCE node
I20260927 16:36:07.348388 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:07.349164 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:07.350355 14727 hybrid_clock.cc:648] HybridClock initialized: now 1790526967350325 us; error 54 us; skew 500 ppm
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:07.353336 14727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:36:07.354457 14727 webserver.cc:460] Webserver started at http://127.0.98.1:33285/ using document root <none> and password file <none>
I20260927 16:36:07.355047 14727 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:07.355118 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:07.359122 14727 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:36:07.361465 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.362835 14727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:07.362929 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "39a4cf3c60924b4f9990ed57bed7be57"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "774270dc2c957da47e482214aaca1414"
server_key_iv: "b5978010df28f0baa3b196ad7bcb8854"
server_key_version: "encryptionkey@0"
I20260927 16:36:07.363505 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:07.380290 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:07.383742 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:07.383985 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:07.384719 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:07.385967 14769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:36:07.390353 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:36:07.390424 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 16:36:07.390496 14727 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:36:07.392199 14727 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:36:07.392258 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:36:07.392539 14769 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap starting.
I20260927 16:36:07.403118 14727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37005
I20260927 16:36:07.403141 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37005 every 8 connection(s)
I20260927 16:36:07.404345 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:36:07.404860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14727
I20260927 16:36:07.405339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37137
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=43881
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:07.414077 14769 log.cc:824] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:07.422468 14769 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:07.422956 14769 tablet_bootstrap.cc:492] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap complete.
I20260927 16:36:07.424284 14769 ts_tablet_manager.cc:1397] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20260927 16:36:07.425316 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.406764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37897 (local address 127.0.98.62:43815)
0927 16:36:07.407298 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.407321 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.407844 (+ 523us) server_negotiation.cc:407] Connection header received
0927 16:36:07.409063 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.409086 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.409559 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.409930 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:07.411529 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.413195 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.415163 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.415831 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.415996 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.419374 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:07.419381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:07.419397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:07.419431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:07.421482 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.422118 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.422124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.422130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.422333 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.422714 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.422719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.422722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.423224 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:07.423366 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.424036 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.424374 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":109,"threads_started":1}
I20260927 16:36:07.425333 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.406762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:37897)
0927 16:36:07.407629 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.407686 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.408580 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.410026 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.410038 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.410595 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:07.411309 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.411333 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.413355 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.413361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.414992 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.415007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.415990 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.415994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.416209 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.416932 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:07.416955 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:07.419147 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:07.421609 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.421614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.421629 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.421972 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.422435 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.422438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.422440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.422611 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.423347 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:07.423354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:07.423679 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.423877 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.424247 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":138,"threads_started":1}
I20260927 16:36:07.427217 14769 raft_consensus.cc:348] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.427766 14769 raft_consensus.cc:729] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a4cf3c60924b4f9990ed57bed7be57, State: Initialized, Role: FOLLOWER
I20260927 16:36:07.428293 14769 consensus_queue.cc:260] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.428694 14877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 16:36:07.428923 14877 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:07.429378 14769 ts_tablet_manager.cc:1428] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:36:07.429486 14877 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:07.432127 14689 ts_manager.cc:194] Registered new tserver with Master: 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:07.436169 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37897
I20260927 16:36:07.438684 14877 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:36:07.446502 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.435508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.0.1:34600)
0927 16:36:07.436234 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.436269 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.436440 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.436915 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.436923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.437405 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:07.437534 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.437543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.438967 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.438970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.439710 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.439719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.440327 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.440342 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.440510 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.441068 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:07.441080 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:07.442669 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:07.444801 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.444808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.444812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.445025 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.445357 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.445359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.445361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.445433 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.445965 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:07.445969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:07.446062 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.446234 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.446376 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":137,"threads_started":1}
I20260927 16:36:07.446637 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.435558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.98.1:37005)
0927 16:36:07.435968 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.435979 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.436551 (+ 572us) server_negotiation.cc:407] Connection header received
0927 16:36:07.436612 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.436615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.436770 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.437041 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:07.437672 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.438827 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.439833 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.440209 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.440315 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.442800 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:07.442810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:07.442824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:07.442852 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:07.444684 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.445141 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.445150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.445153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.445265 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.445520 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.445523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.445525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.445850 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:07.445994 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.446312 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.446474 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":105,"threads_started":1}
I20260927 16:36:07.447518 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a8fee875654450ad1dfe8df92cece6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260927 16:36:07.448325 14886 tablet_replica.cc:330] stopping tablet replica
I20260927 16:36:07.448603 14886 raft_consensus.cc:2229] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:07.448848 14886 raft_consensus.cc:2258] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:07.449770 14886 ts_tablet_manager.cc:1905] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:07.453425 14886 ts_tablet_manager.cc:1918] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:07.453536 14886 log.cc:1196] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/f5a8fee875654450ad1dfe8df92cece6
I20260927 16:36:07.453809 14886 ts_tablet_manager.cc:1939] T f5a8fee875654450ad1dfe8df92cece6 P 39a4cf3c60924b4f9990ed57bed7be57: Deleting consensus metadata
I20260927 16:36:07.455439 14674 catalog_manager.cc:4841] TS 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005): tablet f5a8fee875654450ad1dfe8df92cece6 (table test-table [id=e4ce627b1fb34f019f61b72df6595c98]) successfully deleted
W20260927 16:36:07.529829 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:07.530124 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:07.530223 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:07.534219 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:07.534309 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:07.534403 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:36:07.539619 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43881
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:07.540724 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:07.541754 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:36:07.552403 14879 server_base.cc:1034] running on GCE node
W20260927 16:36:07.552451 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.552335 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.552363 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:07.552974 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:07.553670 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:07.554858 14879 hybrid_clock.cc:648] HybridClock initialized: now 1790526967554839 us; error 30 us; skew 500 ppm
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:07.557571 14879 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:36:07.558550 14879 webserver.cc:460] Webserver started at http://127.0.98.2:43881/ using document root <none> and password file <none>
I20260927 16:36:07.559113 14879 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:07.559185 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:07.562987 14879 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:07.564935 14901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.566097 14879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:36:07.566263 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "21479876f0e74df984203cd6e36a4f28"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "7e8356b2e30cf47bcd5fbb2c3f6c9a59"
server_key_iv: "5d202432e59db1c06870600af0698a61"
server_key_version: "encryptionkey@0"
I20260927 16:36:07.566877 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:07.587741 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:07.590936 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:07.591150 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:07.591777 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:07.592654 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:07.592726 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.592814 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:07.592854 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.602342 14879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37137
I20260927 16:36:07.602361 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37137 every 8 connection(s)
I20260927 16:36:07.603447 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:07.613225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 14879
I20260927 16:36:07.613770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43757
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=45659
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:07.617189 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.605216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44631 (local address 127.0.98.62:43815)
0927 16:36:07.605368 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.605372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.606430 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 16:36:07.607321 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.607324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.607388 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.607498 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:07.608682 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.609210 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.610370 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.610604 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.610726 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.613181 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:07.613186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:07.613190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:07.613223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:07.615074 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.615588 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.615593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.615595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.615685 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.616027 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.616031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.616033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.616219 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:07.616332 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.616940 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.617063 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:36:07.618489 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.605592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:44631)
0927 16:36:07.606279 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.606322 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.607104 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.607677 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.607685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.608037 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:07.608512 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.608525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.609339 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.609343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.610240 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.610250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.610763 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.610767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.610938 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.611536 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:07.611552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:07.612998 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:07.615207 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.615212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.615224 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.615471 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.615789 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.615797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.615799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.615927 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.616330 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:07.616336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:07.616671 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.616884 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.617238 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":101,"threads_started":1}
I20260927 16:36:07.619728 15015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 16:36:07.620016 15015 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:07.620671 15015 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:07.621853 14689 ts_manager.cc:194] Registered new tserver with Master: 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
I20260927 16:36:07.622536 14689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44631
W20260927 16:36:07.728976 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:07.729267 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:07.729341 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:07.733301 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:07.733399 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:07.733495 15018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:36:07.738611 15018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45659
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:07.739661 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:07.740684 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:07.750752 15024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:07.750748 15026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:07.750804 15018 server_base.cc:1034] running on GCE node
W20260927 16:36:07.750741 15023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:07.751441 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:07.752120 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:07.753302 15018 hybrid_clock.cc:648] HybridClock initialized: now 1790526967753265 us; error 50 us; skew 500 ppm
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:07.756024 15018 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:36:07.756834 15018 webserver.cc:460] Webserver started at http://127.0.98.3:45659/ using document root <none> and password file <none>
I20260927 16:36:07.757350 15018 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:07.757421 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:07.761219 15018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:36:07.763204 15033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.764444 15018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:36:07.764537 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "e5e426b8769c4d0eb1e71f9ba682d66c"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "dd72343c453ca7dcb130b73cf893064b"
server_key_iv: "6134c61875ca193522eabe727a4b3b69"
server_key_version: "encryptionkey@0"
I20260927 16:36:07.765043 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:07.778473 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:07.781245 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:07.781400 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:07.781931 15018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:07.782802 15018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:07.782852 15018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.782893 15018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:07.782908 15018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:07.792276 15018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43757
I20260927 16:36:07.792299 15146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43757 every 8 connection(s)
I20260927 16:36:07.793345 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:36:07.800900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15018
Sep 27 16:36:07 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:07.807109 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.795174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40515 (local address 127.0.98.62:43815)
0927 16:36:07.795315 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.795319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.796355 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 16:36:07.797326 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.797328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.797375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.797452 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:07.798650 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.799166 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.800235 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.800441 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.800516 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.803508 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:07.803511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:07.803513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:07.803535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:07.805012 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.805525 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.805530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.805532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.805615 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.805956 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.805961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.805963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.806129 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:07.806280 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.806775 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.806938 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:36:07.807953 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.795573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:40515)
0927 16:36:07.796230 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.796270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.797100 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.797642 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.797651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.798011 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:07.798524 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.798537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.799271 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.799273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.800122 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.800130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.800562 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.800564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.800699 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.801339 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:07.801355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:07.803332 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:07.805148 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.805154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.805169 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.805423 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.805715 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:07.805718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:07.805720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:07.805860 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:07.806275 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:07.806282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:07.806529 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.806718 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.807046 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":122,"threads_started":1}
I20260927 16:36:07.809216 15147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 16:36:07.809480 15147 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:07.809845 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.801581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.98.62:43815)
0927 16:36:07.801824 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.801829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.801939 (+ 110us) server_negotiation.cc:407] Connection header received
0927 16:36:07.802025 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.802031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.802088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.802200 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:07.803106 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.803803 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.804639 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.804868 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.804973 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.806310 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:07.806320 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:07.806322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:07.806351 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:07.808027 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.808552 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.808558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.808561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.808647 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:07.808967 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:07.808970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:07.808971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:07.809120 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:07.809352 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.809562 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.809699 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260927 16:36:07.810057 15147 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 16:36:07.811192 14688 ts_manager.cc:194] Registered new tserver with Master: e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757)
I20260927 16:36:07.811761 14688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40515
I20260927 16:36:07.823405 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:36:07.827725 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.824038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.98.62:43815)
0927 16:36:07.824176 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.824179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.824385 (+ 206us) server_negotiation.cc:407] Connection header received
0927 16:36:07.824527 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.824530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.824639 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.824737 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:07.824951 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.825469 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.826395 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.826623 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.826710 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.826974 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:07.827401 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:36:07.827492 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.827553 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.827607 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:36:07.828404 14688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:36:07.831688 14688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:36:07.839047 14811 tablet_service.cc:1467] Processing CreateTablet for tablet 2e95551b255a47c28538fe00f1e89b90 (DEFAULT_TABLE table=test-table [id=6a211759a6384a9a945b2a4c447579db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:07.839705 14811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e95551b255a47c28538fe00f1e89b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:07.842475 14769 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap starting.
I20260927 16:36:07.843365 14769 tablet_bootstrap.cc:654] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:07.844025 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.837978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43757 (local address 127.0.0.1:48368)
0927 16:36:07.838156 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.838171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.838262 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.839365 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.839369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.839412 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:07.839542 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.839549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.841003 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.841009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.842468 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.842479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.843582 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.843586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.843676 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.843716 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.843770 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.843866 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":115}
I20260927 16:36:07.844327 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.838233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37137 (local address 127.0.0.1:51402)
0927 16:36:07.838610 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.838623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.838706 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.839442 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.839445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.839457 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:07.839546 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.839550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.841131 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.841134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.842468 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.842479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.843604 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.843609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.843741 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.843778 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.843870 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.843944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":76,"thread_start_us":100,"threads_started":1}
I20260927 16:36:07.844844 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.838670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.0.98.2:37137)
0927 16:36:07.838981 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.838992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.839016 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:36:07.839095 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.839100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.839303 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.839468 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:07.839628 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.841003 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.842614 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.843494 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.843599 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.843768 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.844585 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.844699 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260927 16:36:07.844882 14769 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: No bootstrap required, opened a new log
I20260927 16:36:07.844960 14769 ts_tablet_manager.cc:1397] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:36:07.844980 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.838396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.3:43757)
0927 16:36:07.838889 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.838901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.838926 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:36:07.839004 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.839008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.839176 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.839707 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:07.839877 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.840866 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.842613 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.843441 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.844542 (+ 1101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.844706 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.844759 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.844825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":134,"threads_started":1}
I20260927 16:36:07.845382 14769 raft_consensus.cc:348] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.845504 14769 raft_consensus.cc:374] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:07.845541 14769 raft_consensus.cc:729] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a4cf3c60924b4f9990ed57bed7be57, State: Initialized, Role: FOLLOWER
I20260927 16:36:07.845662 14769 consensus_queue.cc:260] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.845937 14769 ts_tablet_manager.cc:1428] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:36:07.847900 15081 tablet_service.cc:1467] Processing CreateTablet for tablet 2e95551b255a47c28538fe00f1e89b90 (DEFAULT_TABLE table=test-table [id=6a211759a6384a9a945b2a4c447579db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:07.847900 14949 tablet_service.cc:1467] Processing CreateTablet for tablet 2e95551b255a47c28538fe00f1e89b90 (DEFAULT_TABLE table=test-table [id=6a211759a6384a9a945b2a4c447579db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:07.849002 15081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e95551b255a47c28538fe00f1e89b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:07.848999 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e95551b255a47c28538fe00f1e89b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:07.857101 15161 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Bootstrap starting.
I20260927 16:36:07.857539 15160 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Bootstrap starting.
I20260927 16:36:07.859503 15161 tablet_bootstrap.cc:654] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:07.860090 15160 tablet_bootstrap.cc:654] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:07.860836 15160 log.cc:824] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:07.860836 15161 log.cc:824] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:07.862989 15161 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: No bootstrap required, opened a new log
I20260927 16:36:07.863198 15161 ts_tablet_manager.cc:1397] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:36:07.863487 15160 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: No bootstrap required, opened a new log
I20260927 16:36:07.863662 15160 ts_tablet_manager.cc:1397] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:36:07.867424 15161 raft_consensus.cc:348] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.867797 15161 raft_consensus.cc:374] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:07.867580 15160 raft_consensus.cc:348] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.867856 15161 raft_consensus.cc:729] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e426b8769c4d0eb1e71f9ba682d66c, State: Initialized, Role: FOLLOWER
I20260927 16:36:07.867893 15160 raft_consensus.cc:374] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:07.867939 15160 raft_consensus.cc:729] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21479876f0e74df984203cd6e36a4f28, State: Initialized, Role: FOLLOWER
I20260927 16:36:07.868412 15161 consensus_queue.cc:260] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.868412 15160 consensus_queue.cc:260] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.869544 15160 ts_tablet_manager.cc:1428] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:36:07.869544 15161 ts_tablet_manager.cc:1428] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:36:07.869925 15015 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 16:36:07.869925 15147 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 16:36:07.878579 15165 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:07.878763 15165 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.879979 15165 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137), 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:07.883690 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.880193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.98.3:58217)
0927 16:36:07.880297 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.880316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.880401 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.880737 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.880740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.880760 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:07.880882 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.880888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.881663 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.881669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.882579 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.882589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.883296 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.883300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.883397 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.883418 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.883482 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.883556 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 16:36:07.883690 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.880315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58217 (local address 127.0.98.1:37005)
0927 16:36:07.880423 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.880428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.880448 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:36:07.880522 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.880525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.880595 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.880713 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:07.880997 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.881550 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.882709 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.883170 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.883298 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.883472 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.883510 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.883577 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 16:36:07.883932 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.880433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59433 (local address 127.0.98.2:37137)
0927 16:36:07.880559 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.880564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.880869 (+ 305us) server_negotiation.cc:407] Connection header received
0927 16:36:07.881006 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.881009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.881058 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.881133 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:07.881398 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.881889 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.883000 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.883403 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.883520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.883578 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.883759 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.883825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:36:07.883956 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.880192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37137 (local address 127.0.98.3:59433)
0927 16:36:07.880771 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.880782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.880899 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:07.881151 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:07.881153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:07.881175 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:07.881289 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.881294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.882000 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.882006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.882869 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:07.882878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.883522 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.883525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:07.883622 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.883644 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:36:07.883742 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 16:36:07.883808 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":278,"thread_start_us":116,"threads_started":1}
I20260927 16:36:07.884411 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e95551b255a47c28538fe00f1e89b90" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a4cf3c60924b4f9990ed57bed7be57" is_pre_election: true
I20260927 16:36:07.884658 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e95551b255a47c28538fe00f1e89b90" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21479876f0e74df984203cd6e36a4f28" is_pre_election: true
I20260927 16:36:07.884740 14831 raft_consensus.cc:2454] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 0.
I20260927 16:36:07.884938 14969 raft_consensus.cc:2454] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 0.
I20260927 16:36:07.885210 15035 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57, e5e426b8769c4d0eb1e71f9ba682d66c; no voters:
I20260927 16:36:07.885535 15165 raft_consensus.cc:2781] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:36:07.885607 15165 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:36:07.885658 15165 raft_consensus.cc:3037] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:07.886775 15165 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.887171 15165 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 election: Requested vote from peers 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137), 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005)
I20260927 16:36:07.887503 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e95551b255a47c28538fe00f1e89b90" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21479876f0e74df984203cd6e36a4f28"
I20260927 16:36:07.887511 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e95551b255a47c28538fe00f1e89b90" candidate_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a4cf3c60924b4f9990ed57bed7be57"
I20260927 16:36:07.887653 14831 raft_consensus.cc:3037] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:07.887655 14969 raft_consensus.cc:3037] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:07.888819 14831 raft_consensus.cc:2454] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 1.
I20260927 16:36:07.888862 14969 raft_consensus.cc:2454] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e426b8769c4d0eb1e71f9ba682d66c in term 1.
I20260927 16:36:07.889102 15035 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57, e5e426b8769c4d0eb1e71f9ba682d66c; no voters:
I20260927 16:36:07.889297 15165 raft_consensus.cc:2781] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:07.889530 15165 raft_consensus.cc:686] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 LEADER]: Becoming Leader. State: Replica: e5e426b8769c4d0eb1e71f9ba682d66c, State: Running, Role: LEADER
I20260927 16:36:07.889858 15165 consensus_queue.cc:237] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:07.892954 14689 catalog_manager.cc:5495] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } health_report { overall_health: UNKNOWN } } }
I20260927 16:36:07.905452 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:07.902243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.98.3:43757)
0927 16:36:07.902361 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:07.902366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:07.902387 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:36:07.902516 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:07.902518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:07.902571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:07.902659 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:07.902850 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.903345 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.904153 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:07.904365 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:07.904438 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:07.904684 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:07.905115 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:36:07.905221 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:07.905270 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:36:07.905322 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 16:36:07.906714 14878 tablet.cc:2369] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:36:07.912232 14831 raft_consensus.cc:1261] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Refusing update from remote peer e5e426b8769c4d0eb1e71f9ba682d66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:07.912238 14969 raft_consensus.cc:1261] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Refusing update from remote peer e5e426b8769c4d0eb1e71f9ba682d66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:07.912864 15165 consensus_queue.cc:1035] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:07.913049 15167 consensus_queue.cc:1035] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:07.920133 15170 mvcc.cc:205] Tried to move back new op lower bound from 7333998460561362944 to 7333998460479684608. Current Snapshot: MvccSnapshot[applied={T|T < 7333998460561362944}]
I20260927 16:36:07.928045 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:07.928184 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:07.930490 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Sending DeleteTablet for 3 replicas of tablet 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:07.931145 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e95551b255a47c28538fe00f1e89b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260927 16:36:07.931250 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e95551b255a47c28538fe00f1e89b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260927 16:36:07.931351 14886 tablet_replica.cc:330] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: stopping tablet replica
I20260927 16:36:07.931464 14886 raft_consensus.cc:2229] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:07.931610 14886 raft_consensus.cc:2258] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:07.931923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14727
I20260927 16:36:07.931895 15081 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e95551b255a47c28538fe00f1e89b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48368
I20260927 16:36:07.932924 15180 tablet_replica.cc:330] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: stopping tablet replica
I20260927 16:36:07.933240 15180 raft_consensus.cc:2229] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:36:07.933646 15180 raft_consensus.cc:2258] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:07.934679 15179 tablet_replica.cc:330] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: stopping tablet replica
I20260927 16:36:07.935001 15179 raft_consensus.cc:2229] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:07.935269 15180 ts_tablet_manager.cc:1905] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:07.935334 15179 raft_consensus.cc:2258] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:07.936802 15179 ts_tablet_manager.cc:1905] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:07.938064 15180 ts_tablet_manager.cc:1918] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:07.938184 15180 log.cc:1196] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:07.938530 15180 ts_tablet_manager.cc:1939] T 2e95551b255a47c28538fe00f1e89b90 P e5e426b8769c4d0eb1e71f9ba682d66c: Deleting consensus metadata
I20260927 16:36:07.939751 14673 catalog_manager.cc:4841] TS e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): tablet 2e95551b255a47c28538fe00f1e89b90 (table test-table [id=6a211759a6384a9a945b2a4c447579db]) successfully deleted
I20260927 16:36:07.940153 15179 ts_tablet_manager.cc:1918] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:07.940274 15179 log.cc:1196] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:07.940613 15179 ts_tablet_manager.cc:1939] T 2e95551b255a47c28538fe00f1e89b90 P 21479876f0e74df984203cd6e36a4f28: Deleting consensus metadata
W20260927 16:36:07.940781 14674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37005 (error 108)
I20260927 16:36:07.941847 14673 catalog_manager.cc:4841] TS 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): tablet 2e95551b255a47c28538fe00f1e89b90 (table test-table [id=6a211759a6384a9a945b2a4c447579db]) successfully deleted
W20260927 16:36:07.942124 15156 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:36:07.941734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37005 (local address 127.0.0.1:34602)
0927 16:36:07.941876 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:07.941895 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:07.941969 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37005: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 16:36:07.942301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14879
W20260927 16:36:07.942353 14674 catalog_manager.cc:4584] TS 39a4cf3c60924b4f9990ed57bed7be57 (127.0.98.1:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e95551b255a47c28538fe00f1e89b90: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37005: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260927 16:36:07.949400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15018
I20260927 16:36:07.955240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 14656
I20260927 16:36:07.961627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--webserver_interface=127.0.98.62
--webserver_port=45663
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:08.072873 15182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:08.073199 15182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:08.073295 15182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:08.077382 15182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:08.077482 15182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:08.077526 15182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:08.077582 15182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:08.077618 15182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:08.082851 15182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:08.084035 15182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:08.085104 15182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:08.093582 15190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.093591 15187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.094434 15182 server_base.cc:1034] running on GCE node
W20260927 16:36:08.094522 15188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.095124 15182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:08.096238 15182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:08.097429 15182 hybrid_clock.cc:648] HybridClock initialized: now 1790526968097413 us; error 29 us; skew 500 ppm
Sep 27 16:36:08 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:08.100237 15182 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:08.101209 15182 webserver.cc:460] Webserver started at http://127.0.98.62:45663/ using document root <none> and password file <none>
I20260927 16:36:08.101778 15182 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:08.101841 15182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:08.105857 15182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:36:08.108322 15197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.109596 15182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:36:08.109740 15182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "db6fb4e2651640e4806229b12d185d24"
format_stamp: "Formatted at 2026-09-27 16:36:05 on dist-test-slave-t27c"
server_key: "9ccd52adc60c43d85b20de88aa56b918"
server_key_iv: "eba9247882f26a2f0bfc7719c669feaf"
server_key_version: "encryptionkey@0"
I20260927 16:36:08.110406 15182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:08.121707 15182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:08.125013 15182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:08.125169 15182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:08.132473 15182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43815
I20260927 16:36:08.132773 15249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43815 every 8 connection(s)
I20260927 16:36:08.133594 15182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:08.137635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15182
I20260927 16:36:08.137912 15250 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:36:08.138118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37005
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=33285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:08.139693 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap starting.
I20260927 16:36:08.157305 15250 log.cc:824] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:08.168295 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:08.168684 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24: Bootstrap complete.
I20260927 16:36:08.172675 15250 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:08.173097 15250 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Initialized, Role: FOLLOWER
I20260927 16:36:08.173571 15250 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:08.173717 15250 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:08.173807 15250 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:08.173929 15250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:36:08.176349 15250 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:08.176769 15250 leader_election.cc:302] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6fb4e2651640e4806229b12d185d24; no voters:
I20260927 16:36:08.177162 15250 leader_election.cc:288] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:36:08.177625 15255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:36:08.178023 15255 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [term 3 LEADER]: Becoming Leader. State: Replica: db6fb4e2651640e4806229b12d185d24, State: Running, Role: LEADER
I20260927 16:36:08.178625 15250 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:08.178577 15255 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 16:36:08.180346 15254 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:08.180676 15254 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:08.180557 15255 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6fb4e2651640e4806229b12d185d24. Latest consensus state: current_term: 3 leader_uuid: "db6fb4e2651640e4806229b12d185d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6fb4e2651640e4806229b12d185d24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 16:36:08.180982 15255 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6fb4e2651640e4806229b12d185d24 [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:08.181326 15263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:08.188038 15263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:08.188767 15263 catalog_manager.cc:1259] Loaded cluster ID: 3d9c8e1f7a9348d48f5c8c732159c2f7
I20260927 16:36:08.188839 15263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 16:36:08.189854 15263 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f110705b9ad google::LogMessage::Fail() at ??:0
@ 0x7f110705e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f110705b4de google::LogMessage::Flush() at ??:0
@ 0x7f110705ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f111199afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f111199bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1111998579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f11119dc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f11092ad3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f11092adc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f11092af3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f110929e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f110a8e26db start_thread at ??:0
@ 0x7f1105b5961f clone at ??:0
W20260927 16:36:08.296128 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:08.296509 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:08.296581 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:08.303048 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:08.303146 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:08.303260 15252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:36:08.311949 15252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33285
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:08.313571 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:08.314992 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:08.326694 15294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.326694 15291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.326694 15292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.326997 15252 server_base.cc:1034] running on GCE node
I20260927 16:36:08.327469 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:08.328200 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:08.329418 15252 hybrid_clock.cc:648] HybridClock initialized: now 1790526968329395 us; error 34 us; skew 500 ppm
Sep 27 16:36:08 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:08.333217 15252 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:36:08.334148 15252 webserver.cc:460] Webserver started at http://127.0.98.1:33285/ using document root <none> and password file <none>
I20260927 16:36:08.334820 15252 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:08.334883 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:08.339150 15252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:36:08.341710 15301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.343246 15252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:36:08.343403 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "39a4cf3c60924b4f9990ed57bed7be57"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "774270dc2c957da47e482214aaca1414"
server_key_iv: "b5978010df28f0baa3b196ad7bcb8854"
server_key_version: "encryptionkey@0"
I20260927 16:36:08.344082 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:08.367026 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:08.370918 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:08.371150 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:08.371820 15252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:08.373000 15308 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:36:08.378515 15252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:36:08.378687 15252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 16:36:08.378799 15252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:36:08.380510 15252 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:36:08.380585 15252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:36:08.380872 15308 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap starting.
I20260927 16:36:08.392231 15252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37005
I20260927 16:36:08.392436 15415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37005 every 8 connection(s)
I20260927 16:36:08.393596 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:36:08.395526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15252
I20260927 16:36:08.395993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37137
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=43881
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:08.401129 15308 log.cc:824] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:08.408169 15308 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:08.408545 15308 tablet_bootstrap.cc:492] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Bootstrap complete.
I20260927 16:36:08.409734 15308 ts_tablet_manager.cc:1397] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260927 16:36:08.412102 15308 raft_consensus.cc:348] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:08.412518 15308 raft_consensus.cc:729] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a4cf3c60924b4f9990ed57bed7be57, State: Initialized, Role: FOLLOWER
I20260927 16:36:08.413059 15308 consensus_queue.cc:260] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:08.413996 15308 ts_tablet_manager.cc:1428] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260927 16:36:08.468008 15418 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:36:08.396011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:58985)
0927 16:36:08.399531 (+ 3520us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.399579 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:08.400559 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:08.466992 (+ 66433us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3298,"thread_start_us":143,"threads_started":1}
W20260927 16:36:08.468662 15416 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43815: Client connection negotiation failed: client connection to 127.0.98.62:43815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:36:08.469501 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.469196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:54779)
0927 16:36:08.469340 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.469428 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 16:36:08.470216 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.469972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:55867)
0927 16:36:08.470079 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.470136 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 16:36:08.470413 15416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:36:08.528335 15419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:08.528604 15419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:08.528697 15419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:08.532882 15419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:08.532944 15419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:08.533054 15419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:36:08.538419 15419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43881
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:08.539557 15419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:08.540589 15419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:08.550971 15429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.550959 15426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.550949 15427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.551021 15419 server_base.cc:1034] running on GCE node
I20260927 16:36:08.551621 15419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:08.552328 15419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:08.553530 15419 hybrid_clock.cc:648] HybridClock initialized: now 1790526968553487 us; error 55 us; skew 500 ppm
Sep 27 16:36:08 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:08.556349 15419 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:36:08.557281 15419 webserver.cc:460] Webserver started at http://127.0.98.2:43881/ using document root <none> and password file <none>
I20260927 16:36:08.557869 15419 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:08.557962 15419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:08.562959 15419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:36:08.565600 15436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.567023 15419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:36:08.567127 15419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "21479876f0e74df984203cd6e36a4f28"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "7e8356b2e30cf47bcd5fbb2c3f6c9a59"
server_key_iv: "5d202432e59db1c06870600af0698a61"
server_key_version: "encryptionkey@0"
I20260927 16:36:08.567682 15419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:08.593911 15419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:08.597087 15419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:08.597249 15419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:08.597819 15419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:08.598763 15419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:08.598812 15419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.598852 15419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:08.598878 15419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.610114 15419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37137
I20260927 16:36:08.610209 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37137 every 8 connection(s)
I20260927 16:36:08.611230 15419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:36:08.615196 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.613581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:58595)
0927 16:36:08.614350 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.614470 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":125,"threads_started":1}
W20260927 16:36:08.615796 15550 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43815: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
I20260927 16:36:08.616240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15419
I20260927 16:36:08.616533 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.616269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:60495)
0927 16:36:08.616432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.616480 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 16:36:08.616684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43757
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43815
--webserver_port=45659
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:08.617372 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.617126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:35237)
0927 16:36:08.617264 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.617317 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 16:36:08.617610 15550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:36:08.732313 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:08.732597 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:08.732642 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:08.736752 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:08.736822 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:08.736924 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:36:08.742259 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45659
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:08.743358 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:08.744453 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:08.754904 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:08.754907 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.755273 15553 server_base.cc:1034] running on GCE node
W20260927 16:36:08.754907 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:08.755734 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:08.756376 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:08.757584 15553 hybrid_clock.cc:648] HybridClock initialized: now 1790526968757562 us; error 33 us; skew 500 ppm
Sep 27 16:36:08 dist-test-slave-t27c krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:08.760362 15553 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:36:08.761224 15553 webserver.cc:460] Webserver started at http://127.0.98.3:45659/ using document root <none> and password file <none>
I20260927 16:36:08.761792 15553 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:08.761896 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:08.765774 15553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:36:08.767881 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.769102 15553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:36:08.769233 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "e5e426b8769c4d0eb1e71f9ba682d66c"
format_stamp: "Formatted at 2026-09-27 16:36:06 on dist-test-slave-t27c"
server_key: "dd72343c453ca7dcb130b73cf893064b"
server_key_iv: "6134c61875ca193522eabe727a4b3b69"
server_key_version: "encryptionkey@0"
I20260927 16:36:08.769764 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:08.781623 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:08.784719 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:08.784948 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:08.785595 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:08.786747 15553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:08.786825 15553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.786903 15553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:08.786949 15553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:08.797358 15553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43757
I20260927 16:36:08.797434 15681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43757 every 8 connection(s)
I20260927 16:36:08.798733 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:36:08.802788 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.800971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:40171)
0927 16:36:08.801656 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.801813 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":119,"threads_started":1}
W20260927 16:36:08.803341 15682 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43815: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
I20260927 16:36:08.803808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15553
I20260927 16:36:08.804111 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.803832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:55133)
0927 16:36:08.804011 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.804057 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 16:36:08.804853 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:08.804609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:47313)
0927 16:36:08.804759 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:08.804803 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 16:36:08.805027 15682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:36:09.471714 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:09.471115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:39775)
0927 16:36:09.471490 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:09.471570 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260927 16:36:09.618674 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:09.618278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:37335)
0927 16:36:09.618506 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:09.618574 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260927 16:36:09.805960 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:09.805545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:37885)
0927 16:36:09.805801 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:09.805857 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260927 16:36:10.067782 15689 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:10.068043 15689 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:10.069172 15689 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757), 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
Sep 27 16:36:10 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 16:36:10 dist-test-slave-t27c krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 16:36:10.083227 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.069921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37137 (local address 127.0.98.1:38337)
0927 16:36:10.070313 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:10.070327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:10.070427 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:10.072232 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:10.072237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:10.072424 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:10.072860 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:10.072876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.074410 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.074413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:10.075394 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:10.075404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.076106 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.076109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:10.076270 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:10.077018 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:10.077035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:10.078794 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:10.080885 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:10.080887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:10.080889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:10.081105 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:10.081487 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:10.081491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:10.081584 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:10.082222 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:10.082228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:10.082503 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 16:36:10.082671 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 16:36:10.083006 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":235,"spinlock_wait_cycles":18688,"thread_start_us":160,"threads_started":1}
I20260927 16:36:10.083240 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.069668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53067 (local address 127.0.98.3:43757)
0927 16:36:10.069957 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:10.069984 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:10.070034 (+ 50us) server_negotiation.cc:407] Connection header received
0927 16:36:10.070934 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:10.070942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:10.071363 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:10.071616 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:10.073060 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.074200 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.075552 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.076003 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.076177 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:10.078659 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:10.078666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:10.078681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:10.078716 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:10.080655 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081205 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:10.081213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:10.081220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:10.081338 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081719 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:10.081721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:10.081723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:10.082078 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:10.082245 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:10.082796 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 16:36:10.083048 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260927 16:36:10.083266 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.070258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38337 (local address 127.0.98.2:37137)
0927 16:36:10.070557 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:10.070583 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:10.070623 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:36:10.071553 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:10.071561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:10.071930 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:10.072231 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:10.073043 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.074253 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.075515 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.075980 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.076138 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:10.078971 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:10.078975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:10.078986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:10.079011 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:10.080772 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081243 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:10.081248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:10.081252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:10.081357 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081680 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:10.081686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:10.081688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:10.082079 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:10.082250 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:10.082786 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 16:36:10.083063 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":128,"threads_started":1}
I20260927 16:36:10.083271 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.069538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43757 (local address 127.0.98.1:53067)
0927 16:36:10.069813 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:10.069830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:10.069940 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:10.071764 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:10.071770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:10.072364 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:10.072860 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:10.072877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.074369 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.074372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:10.075393 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:10.075403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:10.076152 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:10.076156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:10.076276 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:10.076852 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:10.076868 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:10.078396 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:10.080772 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:10.080778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:10.080790 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:10.081077 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:10.081437 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:10.081443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:10.081448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:10.081598 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:10.082224 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:10.082230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:10.082489 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 16:36:10.082670 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:36:10.083092 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":279,"spinlock_wait_cycles":8448,"thread_start_us":109,"threads_started":1}
W20260927 16:36:10.086081 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
W20260927 16:36:10.086558 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:10.086630 15302 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57; no voters: 21479876f0e74df984203cd6e36a4f28, e5e426b8769c4d0eb1e71f9ba682d66c
I20260927 16:36:10.087044 15689 raft_consensus.cc:2726] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:36:10.472776 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.472496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:36579)
0927 16:36:10.472626 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:10.472684 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:36:10.619782 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.619392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:33327)
0927 16:36:10.619638 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:10.619696 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260927 16:36:10.807061 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:10.806693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:54205)
0927 16:36:10.806926 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:10.806978 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260927 16:36:11.474036 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:11.473595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:40629)
0927 16:36:11.473870 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:11.473937 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260927 16:36:11.606966 15697 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:11.607121 15697 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:11.607539 15697 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757), 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
W20260927 16:36:11.608431 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
W20260927 16:36:11.608604 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:11.608671 15302 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57; no voters: 21479876f0e74df984203cd6e36a4f28, e5e426b8769c4d0eb1e71f9ba682d66c
I20260927 16:36:11.608835 15697 raft_consensus.cc:2726] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:36:11.620872 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:11.620450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:39839)
0927 16:36:11.620705 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:11.620764 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260927 16:36:11.808246 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:11.807763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:57047)
0927 16:36:11.808084 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:11.808142 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
I20260927 16:36:12.475353 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:12.474912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:40639)
0927 16:36:12.475173 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:12.475254 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260927 16:36:12.622105 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:12.621664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:40257)
0927 16:36:12.621944 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:12.622008 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260927 16:36:12.809508 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:12.809101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:53847)
0927 16:36:12.809357 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:12.809420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260927 16:36:13.476765 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:13.476333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:39177)
0927 16:36:13.476620 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:13.476672 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260927 16:36:13.623327 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:13.622930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:43281)
0927 16:36:13.623173 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:13.623231 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260927 16:36:13.810698 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:13.810304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:41005)
0927 16:36:13.810543 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:13.810604 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260927 16:36:14.477998 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:14.477560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:45377)
0927 16:36:14.477830 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:14.477897 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260927 16:36:14.574994 15707 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:14.575130 15707 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:14.575552 15707 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757), 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
W20260927 16:36:14.576367 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
W20260927 16:36:14.576502 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:14.576563 15302 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57; no voters: 21479876f0e74df984203cd6e36a4f28, e5e426b8769c4d0eb1e71f9ba682d66c
I20260927 16:36:14.576773 15707 raft_consensus.cc:2726] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:36:14.624627 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:14.624129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:48341)
0927 16:36:14.624422 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:14.624495 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260927 16:36:14.811882 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:14.811444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:35419)
0927 16:36:14.811734 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:14.811787 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":135,"threads_started":1}
I20260927 16:36:15.479460 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:15.478880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:33039)
0927 16:36:15.479257 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:15.479335 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
I20260927 16:36:15.625845 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:15.625384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:59463)
0927 16:36:15.625697 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:15.625752 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260927 16:36:15.813053 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:15.812580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:48339)
0927 16:36:15.812900 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:15.812959 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":89,"threads_started":1}
I20260927 16:36:16.480903 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:16.480438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:51615)
0927 16:36:16.480734 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:16.480800 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260927 16:36:16.627081 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:16.626640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:40333)
0927 16:36:16.626928 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:16.626984 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260927 16:36:16.814512 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:16.813978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:43201)
0927 16:36:16.814285 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:16.814360 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260927 16:36:17.482076 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:17.481669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:50097)
0927 16:36:17.481912 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:17.481968 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260927 16:36:17.542086 15717 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:17.542287 15717 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:17.542748 15717 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757), 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
W20260927 16:36:17.543536 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
W20260927 16:36:17.543653 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:17.543692 15302 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57; no voters: 21479876f0e74df984203cd6e36a4f28, e5e426b8769c4d0eb1e71f9ba682d66c
I20260927 16:36:17.543828 15717 raft_consensus.cc:2726] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:36:17.628252 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:17.627800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:51889)
0927 16:36:17.628090 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:17.628155 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260927 16:36:17.815843 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:17.815347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:58107)
0927 16:36:17.815672 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:17.815740 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260927 16:36:18.483608 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:18.483141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:41189)
0927 16:36:18.483441 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:18.483507 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260927 16:36:18.629665 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:18.629188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:33777)
0927 16:36:18.629499 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:18.629570 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
I20260927 16:36:18.817063 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:18.816656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:36047)
0927 16:36:18.816918 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:18.816970 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260927 16:36:19.484813 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:19.484416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:48261)
0927 16:36:19.484664 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:19.484716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260927 16:36:19.630813 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:19.630416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:50599)
0927 16:36:19.630657 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:19.630715 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260927 16:36:19.818333 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:19.817840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:54475)
0927 16:36:19.818124 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:19.818236 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260927 16:36:20.486002 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:20.485552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:39851)
0927 16:36:20.485846 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:20.485905 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260927 16:36:20.632107 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:20.631642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:57083)
0927 16:36:20.631914 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:20.631985 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260927 16:36:20.819595 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:20.819115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:55585)
0927 16:36:20.819414 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:20.819479 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260927 16:36:21.487298 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:21.486881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:43933)
0927 16:36:21.487142 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:21.487201 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260927 16:36:21.633404 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:21.632951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:35345)
0927 16:36:21.633239 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:21.633303 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
I20260927 16:36:21.820801 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:21.820375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:47423)
0927 16:36:21.820639 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:21.820703 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260927 16:36:22.488684 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:22.488121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:38683)
0927 16:36:22.488512 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:22.488582 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":99,"threads_started":1}
I20260927 16:36:22.634711 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:22.634282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:55623)
0927 16:36:22.634554 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:22.634611 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260927 16:36:22.821965 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:22.821551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:57861)
0927 16:36:22.821806 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:22.821862 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260927 16:36:23.490229 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:23.489749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:51703)
0927 16:36:23.490021 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:23.490077 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260927 16:36:23.602144 15736 raft_consensus.cc:482] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:23.602388 15736 raft_consensus.cc:504] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e426b8769c4d0eb1e71f9ba682d66c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43757 } } peers { permanent_uuid: "21479876f0e74df984203cd6e36a4f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37137 } } peers { permanent_uuid: "39a4cf3c60924b4f9990ed57bed7be57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37005 } }
I20260927 16:36:23.602869 15736 leader_election.cc:288] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757), 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137)
W20260927 16:36:23.603737 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21479876f0e74df984203cd6e36a4f28 (127.0.98.2:37137): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
W20260927 16:36:23.603904 15302 leader_election.cc:341] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5e426b8769c4d0eb1e71f9ba682d66c (127.0.98.3:43757): Not found: Tablet not found: 2e95551b255a47c28538fe00f1e89b90
I20260927 16:36:23.603968 15302 leader_election.cc:302] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a4cf3c60924b4f9990ed57bed7be57; no voters: 21479876f0e74df984203cd6e36a4f28, e5e426b8769c4d0eb1e71f9ba682d66c
I20260927 16:36:23.604167 15736 raft_consensus.cc:2726] T 2e95551b255a47c28538fe00f1e89b90 P 39a4cf3c60924b4f9990ed57bed7be57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:36:23.636132 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:23.635632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:40183)
0927 16:36:23.635893 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:23.635977 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260927 16:36:23.810659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15252
I20260927 16:36:23.816373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15419
I20260927 16:36:23.821800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15553
2026-09-27T16:36:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19214 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:36:23 dist-test-slave-t27c krb5kdc[15740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:36:23 dist-test-slave-t27c krb5kdc[15740](info): set up 2 sockets
Sep 27 16:36:23 dist-test-slave-t27c krb5kdc[15740](info): commencing operation
krb5kdc: starting...
W20260927 16:36:24.866360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:36:24 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:36:24.990725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:25.103631 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:25.103909 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:25.103948 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:25.107893 15753 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:36:25.107944 15753 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:36:25.107962 15753 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:36:25.107981 15753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:25.107999 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:25.108017 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:25.108033 15753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:25.108055 15753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:25.114051 15753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:25.115201 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:25.116324 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:25.125943 15758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.126006 15753 server_base.cc:1034] running on GCE node
W20260927 16:36:25.125911 15759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.125911 15761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.126753 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:25.127918 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:25.129084 15753 hybrid_clock.cc:648] HybridClock initialized: now 1790526985129068 us; error 29 us; skew 500 ppm
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:25.131937 15753 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:25.132838 15753 webserver.cc:460] Webserver started at http://127.0.98.62:42407/ using document root <none> and password file <none>
I20260927 16:36:25.133409 15753 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:25.133497 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:25.133726 15753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:25.135908 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "8d06a49b60ef44cabdb2887c27cce01f"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "fdcc27ba46353023997737ceb0d67839"
server_key_iv: "10e45e89777c93d5dcf4af4b3d841827"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.136504 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d06a49b60ef44cabdb2887c27cce01f"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "fdcc27ba46353023997737ceb0d67839"
server_key_iv: "10e45e89777c93d5dcf4af4b3d841827"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.140789 15753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:36:25.143677 15768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.145176 15753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 16:36:25.145331 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8d06a49b60ef44cabdb2887c27cce01f"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "fdcc27ba46353023997737ceb0d67839"
server_key_iv: "10e45e89777c93d5dcf4af4b3d841827"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.145452 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:25.165684 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:25.169209 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:25.169420 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:25.176783 15753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43605
I20260927 16:36:25.177094 15820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43605 every 8 connection(s)
I20260927 16:36:25.177847 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:25.180960 15821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:25.186913 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap starting.
I20260927 16:36:25.187238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15753
I20260927 16:36:25.187355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:36:25.187624 392 external_mini_cluster.cc:1428] Setting key d7e60d906c1f1a09b35d1de49afc5213
I20260927 16:36:25.189721 15821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:25.190656 15821 log.cc:824] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:25.192531 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: No bootstrap required, opened a new log
I20260927 16:36:25.196117 15821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:25.196376 15821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:25.196452 15821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Initialized, Role: FOLLOWER
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:25.196982 15821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:25.197135 15821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:25.197232 15821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:25.197337 15821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:25.198622 15821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:25.199024 15821 leader_election.cc:302] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d06a49b60ef44cabdb2887c27cce01f; no voters:
I20260927 16:36:25.199406 15821 leader_election.cc:288] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:36:25.199586 15826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:25.199838 15826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 LEADER]: Becoming Leader. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Running, Role: LEADER
I20260927 16:36:25.200196 15826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:25.200925 15821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:25.201923 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d06a49b60ef44cabdb2887c27cce01f. Latest consensus state: current_term: 1 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:36:25.202049 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:25.202349 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:36:25.202446 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:25.202965 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:25.204133 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.189035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.0.98.62:43605)
0927 16:36:25.189480 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.189499 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.189543 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:36:25.190464 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.190489 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.190918 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.191310 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:25.192092 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.193190 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.194053 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.194624 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.194790 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.197483 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:25.197491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:25.197510 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:25.197551 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:25.199898 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.200435 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.200440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.200447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.200600 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.200946 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.200950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.200951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.201502 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:25.202390 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.202563 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.202932 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260927 16:36:25.206282 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:25.211951 15836 catalog_manager.cc:1347] Generated new cluster ID: 5bfb6f95c9174625bce43cdc50bcc502
I20260927 16:36:25.212049 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:25.224021 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:36:25.224987 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:25.231755 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Generated new TSK 0
I20260927 16:36:25.232508 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:36:25.270435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:25.397068 15847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:25.397369 15847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:25.397414 15847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:25.401618 15847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:25.401690 15847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:25.401753 15847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:36:25.407249 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:25.408280 15847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:25.409272 15847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:25.418855 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.418872 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.418917 15855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.418987 15847 server_base.cc:1034] running on GCE node
I20260927 16:36:25.419488 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:25.420220 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:25.421394 15847 hybrid_clock.cc:648] HybridClock initialized: now 1790526985421375 us; error 30 us; skew 500 ppm
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:25.424211 15847 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:36:25.425104 15847 webserver.cc:460] Webserver started at http://127.0.98.1:46471/ using document root <none> and password file <none>
I20260927 16:36:25.425674 15847 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:25.425736 15847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:25.425972 15847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:25.428243 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a47eba3c5b0481781e66cc47b3577c5"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "da707a725e11ea0d8224e74ca6d014ed"
server_key_iv: "9dda97470d83afe0d3173996e851e664"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.428835 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a47eba3c5b0481781e66cc47b3577c5"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "da707a725e11ea0d8224e74ca6d014ed"
server_key_iv: "9dda97470d83afe0d3173996e851e664"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.432737 15847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 16:36:25.435370 15862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.436697 15847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:36:25.436822 15847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "8a47eba3c5b0481781e66cc47b3577c5"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "da707a725e11ea0d8224e74ca6d014ed"
server_key_iv: "9dda97470d83afe0d3173996e851e664"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.436941 15847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:25.456924 15847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:25.459839 15847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:25.460039 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:25.460630 15847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:25.461436 15847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:25.461483 15847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.461555 15847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:25.461588 15847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.470386 15847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34425
I20260927 16:36:25.470418 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34425 every 8 connection(s)
I20260927 16:36:25.471282 15847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:36:25.477695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15847
I20260927 16:36:25.477829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:36:25.478116 392 external_mini_cluster.cc:1428] Setting key f05a5058743bc027a80ecd668cfa3ec7
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:25.486111 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.473122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51385 (local address 127.0.98.62:43605)
0927 16:36:25.473257 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.473261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.474349 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 16:36:25.475223 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.475226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.475276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.475359 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:25.476696 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.477244 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.478600 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.478955 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.479075 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.482226 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:25.482230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:25.482232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:25.482260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:25.483778 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.484428 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.484430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.484431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.484504 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.484934 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.484936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.484937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.485121 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:25.485193 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.485887 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.485987 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:36:25.487447 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.473476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:51385)
0927 16:36:25.474135 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.474228 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.475008 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.475586 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.475594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.475970 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:25.476542 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.476559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.477388 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.477391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.478466 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.478477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.479115 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.479119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.479302 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.479992 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:25.480009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:25.481991 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:25.483898 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.483905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.483923 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.484321 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.484597 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.484602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.484604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.484826 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.485227 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:25.485236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:25.485593 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.485850 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.486275 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:36:25.488667 15976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:25.488931 15976 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:25.489430 15976 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:36:25.491168 15785 ts_manager.cc:194] Registered new tserver with Master: 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:25.492498 15785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:36:25.505857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:25.616112 15981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:25.616370 15981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:25.616413 15981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:25.620424 15981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:25.620481 15981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:25.620537 15981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:36:25.625856 15981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:25.626902 15981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:25.627924 15981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:36:25.637887 15981 server_base.cc:1034] running on GCE node
W20260927 16:36:25.637820 15987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.637820 15986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.637843 15989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.638504 15981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:25.639169 15981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:25.640347 15981 hybrid_clock.cc:648] HybridClock initialized: now 1790526985640314 us; error 47 us; skew 500 ppm
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:25.643224 15981 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:36:25.644184 15981 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260927 16:36:25.644778 15981 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:25.644867 15981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:25.645093 15981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:25.647267 15981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "47634c73ac79473b856809590bac8bf8"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "3937ef42fe56e0e5f2c19417180e9dbd"
server_key_iv: "5324eca96274cefa6b6b0fd6b8b03108"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.647915 15981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "47634c73ac79473b856809590bac8bf8"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "3937ef42fe56e0e5f2c19417180e9dbd"
server_key_iv: "5324eca96274cefa6b6b0fd6b8b03108"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.651719 15981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:36:25.654029 15996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.655293 15981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:25.655418 15981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "47634c73ac79473b856809590bac8bf8"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "3937ef42fe56e0e5f2c19417180e9dbd"
server_key_iv: "5324eca96274cefa6b6b0fd6b8b03108"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.655529 15981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:25.666545 15981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:25.669209 15981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:25.669409 15981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:25.669961 15981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:25.670782 15981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:25.670830 15981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.670892 15981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:25.670926 15981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.680241 15981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45661
I20260927 16:36:25.680282 16109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45661 every 8 connection(s)
I20260927 16:36:25.681267 15981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:36:25.682118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 15981
I20260927 16:36:25.682282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:36:25.682567 392 external_mini_cluster.cc:1428] Setting key 131dc568d47ccacfd8ebbe3d3224b797
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:36:25.696184 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.683448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43669 (local address 127.0.98.62:43605)
0927 16:36:25.683577 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.683582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.684821 (+ 1239us) server_negotiation.cc:407] Connection header received
0927 16:36:25.685766 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.685772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.685835 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.685921 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:25.687482 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.687993 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.689164 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.689400 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.689510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.692271 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:25.692274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:25.692276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:25.692301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:25.693797 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.694398 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.694401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.694402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.694486 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.694888 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.694892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.694894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.695175 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:25.695287 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.695962 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.696059 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:36:25.697431 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.683919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:43669)
0927 16:36:25.684660 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.684721 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.685508 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.686149 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.686159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.686770 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:25.687351 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.687365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.688123 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.688127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.689031 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.689040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.689552 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.689556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.689727 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.690385 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:25.690402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:25.692088 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:25.693907 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.693912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.693925 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.694257 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.694575 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.694577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.694579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.694781 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.695290 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:25.695298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:25.695655 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.695885 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.696261 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":149,"threads_started":1}
I20260927 16:36:25.698603 16110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:25.698907 16110 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:25.699499 16110 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
I20260927 16:36:25.700711 15785 ts_manager.cc:194] Registered new tserver with Master: 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661)
I20260927 16:36:25.701354 15785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:36:25.712240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:25.823961 16115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:25.824290 16115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:25.824378 16115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:25.828429 16115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:25.828509 16115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:25.828593 16115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:36:25.833961 16115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:25.835057 16115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:25.836167 16115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:25.846251 16120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.846361 16115 server_base.cc:1034] running on GCE node
W20260927 16:36:25.846251 16123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:25.846246 16121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:25.847030 16115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:25.847747 16115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:25.848982 16115 hybrid_clock.cc:648] HybridClock initialized: now 1790526985848956 us; error 42 us; skew 500 ppm
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:25.851768 16115 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:36:25.852592 16115 webserver.cc:460] Webserver started at http://127.0.98.3:44185/ using document root <none> and password file <none>
I20260927 16:36:25.853118 16115 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:25.853173 16115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:25.853346 16115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:36:25.855366 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "a0e5257fa2964130a34c6a90351df14e"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "944c625445d08a97f887f3879902b246"
server_key_iv: "cda383829bfd076d91a01cd5550b463e"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.855876 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a0e5257fa2964130a34c6a90351df14e"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "944c625445d08a97f887f3879902b246"
server_key_iv: "cda383829bfd076d91a01cd5550b463e"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.859445 16115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:36:25.861704 16130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.862903 16115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:36:25.862995 16115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "a0e5257fa2964130a34c6a90351df14e"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "944c625445d08a97f887f3879902b246"
server_key_iv: "cda383829bfd076d91a01cd5550b463e"
server_key_version: "encryptionkey@0"
I20260927 16:36:25.863076 16115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:25.883517 16115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:25.886416 16115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:25.886570 16115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:25.887090 16115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:25.887847 16115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:25.887892 16115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.887931 16115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:25.887952 16115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:25.896355 16115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40837
I20260927 16:36:25.896386 16243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40837 every 8 connection(s)
I20260927 16:36:25.897261 16115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:36:25.898466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16115
I20260927 16:36:25.898583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:36:25.898944 392 external_mini_cluster.cc:1428] Setting key be66487e6ffaa0bdd2add9adb328986c
Sep 27 16:36:25 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:25.910328 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.899154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33249 (local address 127.0.98.62:43605)
0927 16:36:25.899284 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.899289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.900413 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 16:36:25.901410 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.901413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.901458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.901586 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:25.902780 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.903372 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.904459 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.904669 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.904743 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.907116 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:25.907119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:25.907122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:25.907146 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:25.908555 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.908995 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.908997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.908999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.909070 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.909349 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.909351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.909352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.909523 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:25.909597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.910046 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.910146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:36:25.911203 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.899559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:33249)
0927 16:36:25.900264 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.900317 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.901229 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.901728 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.901751 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.902092 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:25.902656 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.902669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.903470 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.903472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.904359 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.904367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.904798 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.904801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.904937 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.905463 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:25.905479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:25.906955 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:25.908649 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.908653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.908666 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.908914 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.909143 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:25.909145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:25.909146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:25.909276 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:25.909603 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:25.909608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:25.909845 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.910001 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.910313 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":134,"threads_started":1}
I20260927 16:36:25.912438 16244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:25.912705 16244 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:25.913234 16244 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
I20260927 16:36:25.914249 15784 ts_manager.cc:194] Registered new tserver with Master: a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
I20260927 16:36:25.914844 15784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33249
I20260927 16:36:25.924594 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:36:25.934134 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.926674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.98.62:43605)
0927 16:36:25.926843 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.926848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.926989 (+ 141us) server_negotiation.cc:407] Connection header received
0927 16:36:25.927154 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.927157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.927207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.927291 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:25.928072 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.928601 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.929454 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.929673 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.929771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.931240 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:25.931244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:25.931247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:25.931273 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:25.932451 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.932871 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.932876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.932877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.932957 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:25.933246 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:25.933249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:25.933251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:25.933467 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:25.933605 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.933840 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.933951 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:36:25.937000 15784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:36:25.940114 15784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:36:25.953727 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.948489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45661 (local address 127.0.0.1:52342)
0927 16:36:25.948812 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.948858 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.948975 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.949548 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.949558 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.949583 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:25.949733 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.949743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.951193 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.951197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.952204 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.952213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.953188 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.953192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.953313 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.953361 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.953447 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.953569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260927 16:36:25.954389 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.948628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.2:45661)
0927 16:36:25.949025 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.949034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.949057 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:36:25.949121 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.949125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.949338 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.949510 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:25.949856 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.951074 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.952332 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.953052 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.953934 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.954082 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.954131 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.954247 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
I20260927 16:36:25.954666 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.949215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40837 (local address 127.0.0.1:44606)
0927 16:36:25.949974 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.949984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.950047 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.950765 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.950769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.950788 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:25.950890 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.950896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.951951 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.951954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.953085 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.953094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.954225 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.954232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.954340 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.954368 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.954456 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.954530 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":29,"thread_start_us":103,"threads_started":1}
I20260927 16:36:25.954921 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.949439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.98.3:40837)
0927 16:36:25.950304 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.950317 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.950340 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:36:25.950410 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.950431 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.950628 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.950771 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:25.951007 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.951816 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.953226 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.953897 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.954001 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.954143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.954669 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.954787 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":157,"threads_started":1}
I20260927 16:36:25.957271 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.949547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.0.1:57076)
0927 16:36:25.950993 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:25.951006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:25.951101 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:25.952948 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:25.952952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:25.952971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:25.953087 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.953097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.954466 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.954469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.955609 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:25.955621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.956848 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.956852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:25.956950 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.956974 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:36:25.957044 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:36:25.957115 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260927 16:36:25.957808 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:25.949964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.98.1:34425)
0927 16:36:25.950304 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:25.950318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:25.951096 (+ 778us) server_negotiation.cc:407] Connection header received
0927 16:36:25.952397 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:25.952403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:25.952579 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:25.952719 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:25.953223 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.954325 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.955750 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:25.956531 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:25.956641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:25.956784 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:25.957555 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 16:36:25.957659 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260927 16:36:25.957654 16178 tablet_service.cc:1467] Processing CreateTablet for tablet 50139fc633d142cfb4cb880a576dc758 (DEFAULT_TABLE table=test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:25.957700 16044 tablet_service.cc:1467] Processing CreateTablet for tablet 50139fc633d142cfb4cb880a576dc758 (DEFAULT_TABLE table=test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:25.958752 16178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50139fc633d142cfb4cb880a576dc758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:25.958840 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50139fc633d142cfb4cb880a576dc758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:25.960788 15910 tablet_service.cc:1467] Processing CreateTablet for tablet 50139fc633d142cfb4cb880a576dc758 (DEFAULT_TABLE table=test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:25.961828 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50139fc633d142cfb4cb880a576dc758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:25.964946 16261 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Bootstrap starting.
I20260927 16:36:25.966228 16262 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Bootstrap starting.
I20260927 16:36:25.968145 16261 tablet_bootstrap.cc:654] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:25.968920 16262 tablet_bootstrap.cc:654] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:25.969075 16263 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap starting.
I20260927 16:36:25.969105 16261 log.cc:824] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:25.969697 16262 log.cc:824] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:25.971156 16263 tablet_bootstrap.cc:654] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:25.971477 16261 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: No bootstrap required, opened a new log
I20260927 16:36:25.971671 16261 ts_tablet_manager.cc:1397] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 16:36:25.971683 16262 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: No bootstrap required, opened a new log
I20260927 16:36:25.971874 16262 ts_tablet_manager.cc:1397] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:36:25.972359 16263 log.cc:824] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:25.974476 16263 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: No bootstrap required, opened a new log
I20260927 16:36:25.974651 16263 ts_tablet_manager.cc:1397] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:36:25.975155 16261 raft_consensus.cc:348] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.975409 16261 raft_consensus.cc:374] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:25.975468 16261 raft_consensus.cc:729] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47634c73ac79473b856809590bac8bf8, State: Initialized, Role: FOLLOWER
I20260927 16:36:25.975636 16262 raft_consensus.cc:348] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.975901 16262 raft_consensus.cc:374] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:25.975955 16262 raft_consensus.cc:729] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e5257fa2964130a34c6a90351df14e, State: Initialized, Role: FOLLOWER
I20260927 16:36:25.975920 16261 consensus_queue.cc:260] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.976361 16262 consensus_queue.cc:260] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.976533 16110 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
I20260927 16:36:25.976887 16261 ts_tablet_manager.cc:1428] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:36:25.977276 16262 ts_tablet_manager.cc:1428] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 16:36:25.977581 16244 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
I20260927 16:36:25.979113 16263 raft_consensus.cc:348] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.979324 16263 raft_consensus.cc:374] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:25.979384 16263 raft_consensus.cc:729] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a47eba3c5b0481781e66cc47b3577c5, State: Initialized, Role: FOLLOWER
I20260927 16:36:25.980363 16263 consensus_queue.cc:260] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:25.980958 15976 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
I20260927 16:36:25.981389 16263 ts_tablet_manager.cc:1428] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:36:26.067421 16268 raft_consensus.cc:482] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:26.067653 16268 raft_consensus.cc:504] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:26.068815 16268 leader_election.cc:288] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:26.072779 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.069106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45661 (local address 127.0.98.3:43319)
0927 16:36:26.069252 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:26.069271 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:26.069398 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:26.069715 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:26.069718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:26.069735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:26.069858 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:26.069863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.070649 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.070654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:26.071593 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:26.071605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.072449 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.072451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:26.072535 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:26.072556 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:36:26.072623 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:36:26.072678 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 16:36:26.072862 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.069195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43319 (local address 127.0.98.2:45661)
0927 16:36:26.069359 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:26.069364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:26.069383 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:36:26.069517 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:26.069522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:26.069587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:26.069677 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:26.069970 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.070537 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.071722 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.072340 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.072439 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:26.072504 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:26.072661 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 16:36:26.072732 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:36:26.073498 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.069427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.98.3:50727)
0927 16:36:26.069795 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:26.069806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:26.069916 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:26.070151 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:26.070154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:26.070208 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:26.070329 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:26.070334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.071025 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.071030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:26.072060 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:26.072071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.073075 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.073080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:26.073180 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:26.073203 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:36:26.073299 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:36:26.073365 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260927 16:36:26.073498 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.069570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50727 (local address 127.0.98.1:34425)
0927 16:36:26.069710 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:26.069714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:26.069884 (+ 170us) server_negotiation.cc:407] Connection header received
0927 16:36:26.070005 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:26.070009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:26.070056 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:26.070131 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:26.070436 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.070915 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.072200 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.072813 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.072895 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:26.072960 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:26.073319 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 16:36:26.073394 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:36:26.073450 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50139fc633d142cfb4cb880a576dc758" candidate_uuid: "a0e5257fa2964130a34c6a90351df14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47634c73ac79473b856809590bac8bf8" is_pre_election: true
I20260927 16:36:26.073787 16064 raft_consensus.cc:2454] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e5257fa2964130a34c6a90351df14e in term 0.
I20260927 16:36:26.074105 15930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50139fc633d142cfb4cb880a576dc758" candidate_uuid: "a0e5257fa2964130a34c6a90351df14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47eba3c5b0481781e66cc47b3577c5" is_pre_election: true
I20260927 16:36:26.074378 16131 leader_election.cc:302] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e; no voters:
I20260927 16:36:26.074460 15930 raft_consensus.cc:2454] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e5257fa2964130a34c6a90351df14e in term 0.
I20260927 16:36:26.074653 16268 raft_consensus.cc:2781] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:36:26.074738 16268 raft_consensus.cc:482] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:36:26.074796 16268 raft_consensus.cc:3037] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:26.075860 16268 raft_consensus.cc:504] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:26.076210 16268 leader_election.cc:288] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [CANDIDATE]: Term 1 election: Requested vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:26.076656 15930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50139fc633d142cfb4cb880a576dc758" candidate_uuid: "a0e5257fa2964130a34c6a90351df14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47eba3c5b0481781e66cc47b3577c5"
I20260927 16:36:26.076658 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50139fc633d142cfb4cb880a576dc758" candidate_uuid: "a0e5257fa2964130a34c6a90351df14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47634c73ac79473b856809590bac8bf8"
I20260927 16:36:26.076797 16064 raft_consensus.cc:3037] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:26.076797 15930 raft_consensus.cc:3037] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:26.077904 15930 raft_consensus.cc:2454] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e5257fa2964130a34c6a90351df14e in term 1.
I20260927 16:36:26.077905 16064 raft_consensus.cc:2454] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e5257fa2964130a34c6a90351df14e in term 1.
I20260927 16:36:26.078271 16131 leader_election.cc:302] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5, a0e5257fa2964130a34c6a90351df14e; no voters:
I20260927 16:36:26.078450 16268 raft_consensus.cc:2781] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:26.078709 16268 raft_consensus.cc:686] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 1 LEADER]: Becoming Leader. State: Replica: a0e5257fa2964130a34c6a90351df14e, State: Running, Role: LEADER
I20260927 16:36:26.079099 16268 consensus_queue.cc:237] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:26.082332 15785 catalog_manager.cc:5495] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e5257fa2964130a34c6a90351df14e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a0e5257fa2964130a34c6a90351df14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260927 16:36:26.109642 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.106545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.0.98.3:40837)
0927 16:36:26.106672 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:26.106676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:26.106688 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:36:26.106762 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:26.106764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:26.106827 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:26.106919 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:26.107123 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.107612 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.108404 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:26.108605 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:26.108678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:26.108937 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:26.109351 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:36:26.109442 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:26.109487 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 16:36:26.109535 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:36:26.115404 16064 raft_consensus.cc:1261] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Refusing update from remote peer a0e5257fa2964130a34c6a90351df14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:26.115404 15930 raft_consensus.cc:1261] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Refusing update from remote peer a0e5257fa2964130a34c6a90351df14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:26.116173 16271 consensus_queue.cc:1035] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:26.116375 16268 consensus_queue.cc:1035] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:26.135416 15785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:26.135612 15785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:26.138073 15785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Sending DeleteTablet for 3 replicas of tablet 50139fc633d142cfb4cb880a576dc758
I20260927 16:36:26.138866 16044 tablet_service.cc:1514] Processing DeleteTablet for tablet 50139fc633d142cfb4cb880a576dc758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52342
I20260927 16:36:26.139098 15910 tablet_service.cc:1514] Processing DeleteTablet for tablet 50139fc633d142cfb4cb880a576dc758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57076
I20260927 16:36:26.139420 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet 50139fc633d142cfb4cb880a576dc758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44606
I20260927 16:36:26.139633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15847
I20260927 16:36:26.140828 16284 tablet_replica.cc:330] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: stopping tablet replica
I20260927 16:36:26.141093 16286 tablet_replica.cc:330] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: stopping tablet replica
I20260927 16:36:26.141168 16284 raft_consensus.cc:2229] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:26.141378 16286 raft_consensus.cc:2229] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:36:26.141489 16284 raft_consensus.cc:2258] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:26.141754 16286 raft_consensus.cc:2258] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:26.143208 16284 ts_tablet_manager.cc:1905] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:26.143260 16286 ts_tablet_manager.cc:1905] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:26.146037 16284 ts_tablet_manager.cc:1918] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:26.146139 16284 log.cc:1196] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/50139fc633d142cfb4cb880a576dc758
I20260927 16:36:26.146490 16284 ts_tablet_manager.cc:1939] T 50139fc633d142cfb4cb880a576dc758 P 47634c73ac79473b856809590bac8bf8: Deleting consensus metadata
I20260927 16:36:26.146620 16286 ts_tablet_manager.cc:1918] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:26.146721 16286 log.cc:1196] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/50139fc633d142cfb4cb880a576dc758
I20260927 16:36:26.147146 16286 ts_tablet_manager.cc:1939] T 50139fc633d142cfb4cb880a576dc758 P a0e5257fa2964130a34c6a90351df14e: Deleting consensus metadata
I20260927 16:36:26.147773 15769 catalog_manager.cc:4841] TS 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): tablet 50139fc633d142cfb4cb880a576dc758 (table test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]) successfully deleted
W20260927 16:36:26.148490 15769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34425 (error 108)
I20260927 16:36:26.148751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15981
I20260927 16:36:26.148979 15769 catalog_manager.cc:4841] TS a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): tablet 50139fc633d142cfb4cb880a576dc758 (table test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]) successfully deleted
I20260927 16:36:26.151057 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:26.150755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.0.1:57082)
0927 16:36:26.150896 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:26.150972 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:36:26.151224 15769 catalog_manager.cc:4584] TS 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50139fc633d142cfb4cb880a576dc758: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34425: connect: Connection refused (error 111)
I20260927 16:36:26.155359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16115
I20260927 16:36:26.160516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 15753
I20260927 16:36:46.166196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--webserver_interface=127.0.98.62
--webserver_port=42407
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:36:46.284246 16288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:46.284528 16288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:46.284574 16288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:46.288736 16288 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:36:46.288815 16288 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:36:46.288831 16288 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:36:46.288846 16288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:36:46.288866 16288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:46.288975 16288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:46.289004 16288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:36:46.289037 16288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:36:46.294899 16288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:46.296059 16288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:46.297129 16288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:46.310297 16295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.310343 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.310397 16288 server_base.cc:1034] running on GCE node
W20260927 16:36:46.310328 16294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.311157 16288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:46.312379 16288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:46.313544 16288 hybrid_clock.cc:648] HybridClock initialized: now 1790527006313529 us; error 30 us; skew 500 ppm
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:46.322656 16288 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:36:46.323705 16288 webserver.cc:460] Webserver started at http://127.0.98.62:42407/ using document root <none> and password file <none>
I20260927 16:36:46.324345 16288 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:46.324460 16288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:46.328653 16288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:36:46.331306 16304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.332836 16288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:36:46.332999 16288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8d06a49b60ef44cabdb2887c27cce01f"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "fdcc27ba46353023997737ceb0d67839"
server_key_iv: "10e45e89777c93d5dcf4af4b3d841827"
server_key_version: "encryptionkey@0"
I20260927 16:36:46.333695 16288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:46.356546 16288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:46.359974 16288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:46.360186 16288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:46.367674 16288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43605
I20260927 16:36:46.367961 16356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43605 every 8 connection(s)
I20260927 16:36:46.368888 16288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:36:46.373385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16288
I20260927 16:36:46.373404 16357 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:36:46.373844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34425
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=46471
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:36:46.375218 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap starting.
I20260927 16:36:46.392225 16357 log.cc:824] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:46.401041 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:46.401463 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap complete.
I20260927 16:36:46.405653 16357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:46.406088 16357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Initialized, Role: FOLLOWER
I20260927 16:36:46.406642 16357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:46.406788 16357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:36:46.406888 16357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:36:46.407004 16357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:36:46.408970 16357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:46.409327 16357 leader_election.cc:302] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d06a49b60ef44cabdb2887c27cce01f; no voters:
I20260927 16:36:46.409641 16357 leader_election.cc:288] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:36:46.409786 16361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:36:46.410046 16361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 LEADER]: Becoming Leader. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Running, Role: LEADER
I20260927 16:36:46.410450 16361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:36:46.410974 16357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:36:46.411461 16361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d06a49b60ef44cabdb2887c27cce01f. Latest consensus state: current_term: 2 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:36:46.411579 16361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:46.412241 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:36:46.412459 16365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:36:46.412443 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:36:46.417338 16365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:36:46.418033 16365 catalog_manager.cc:1259] Loaded cluster ID: 5bfb6f95c9174625bce43cdc50bcc502
I20260927 16:36:46.418098 16365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:36:46.420599 16365 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:36:46.421312 16365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Loaded TSK: 0
I20260927 16:36:46.433423 16365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Generated new TSK 1
I20260927 16:36:46.433666 16365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:36:46.489717 16359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:46.490022 16359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:46.490096 16359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:46.494331 16359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:46.494422 16359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:46.494519 16359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:36:46.500010 16359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46471
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:46.501062 16359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:46.502076 16359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:46.512648 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.512657 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.512804 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.513441 16359 server_base.cc:1034] running on GCE node
I20260927 16:36:46.513922 16359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:46.514604 16359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:46.515815 16359 hybrid_clock.cc:648] HybridClock initialized: now 1790527006515788 us; error 45 us; skew 500 ppm
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:46.518814 16359 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:36:46.519917 16359 webserver.cc:460] Webserver started at http://127.0.98.1:46471/ using document root <none> and password file <none>
I20260927 16:36:46.520615 16359 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:46.520723 16359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:46.525732 16359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 16:36:46.528286 16394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.529722 16359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:36:46.529882 16359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "8a47eba3c5b0481781e66cc47b3577c5"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "da707a725e11ea0d8224e74ca6d014ed"
server_key_iv: "9dda97470d83afe0d3173996e851e664"
server_key_version: "encryptionkey@0"
I20260927 16:36:46.530537 16359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:46.547709 16359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:46.551056 16359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:46.551266 16359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:46.551852 16359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:46.552951 16401 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:36:46.557904 16359 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:36:46.557983 16359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 16:36:46.558064 16359 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:36:46.559708 16359 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:36:46.559778 16359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:36:46.560052 16401 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap starting.
I20260927 16:36:46.570641 16359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34425
I20260927 16:36:46.570673 16508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34425 every 8 connection(s)
I20260927 16:36:46.571640 16359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:36:46.580488 16401 log.cc:824] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:46.580678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16359
I20260927 16:36:46.581174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45661
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:46.588609 16401 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:36:46.589003 16401 tablet_bootstrap.cc:492] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap complete.
I20260927 16:36:46.590359 16401 ts_tablet_manager.cc:1397] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.012s
I20260927 16:36:46.590400 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.573964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45287 (local address 127.0.98.62:43605)
0927 16:36:46.574418 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.574439 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.574723 (+ 284us) server_negotiation.cc:407] Connection header received
0927 16:36:46.575862 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.575888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.576299 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.576712 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.577810 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.579497 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.580849 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.581577 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.581759 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.584541 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.584546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.584557 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.584584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.586667 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.587327 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.587333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.587339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.587484 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.587913 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.587916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.587918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.588335 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.588473 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.589163 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.589469 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260927 16:36:46.590495 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.573889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:45287)
0927 16:36:46.574537 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:46.574591 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:46.575401 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:46.576746 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:46.576753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:46.577096 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:46.577627 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.577640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.579664 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.579667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.580705 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.580717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.581731 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.581735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.581923 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.582625 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:46.582645 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:46.584344 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:46.586798 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.586807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.586825 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.587199 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.587596 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.587601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.587604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.587794 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.588466 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:46.588475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:46.588807 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 16:36:46.589029 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 16:36:46.589402 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":119,"threads_started":1}
I20260927 16:36:46.593024 16509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:46.593245 16509 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:46.592900 16401 raft_consensus.cc:348] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:46.593425 16401 raft_consensus.cc:729] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a47eba3c5b0481781e66cc47b3577c5, State: Initialized, Role: FOLLOWER
I20260927 16:36:46.593765 16509 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
I20260927 16:36:46.593876 16401 consensus_queue.cc:260] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:46.594992 16401 ts_tablet_manager.cc:1428] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260927 16:36:46.596464 16321 ts_manager.cc:194] Registered new tserver with Master: 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:46.600287 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45287
I20260927 16:36:46.602665 16509 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:36:46.610538 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.599617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.0.1:43238)
0927 16:36:46.600270 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:46.600304 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:46.600460 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:46.600903 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:46.600908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:46.601313 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:46.601443 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.601451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.602932 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.602936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.603921 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.603932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.604551 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.604567 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.604677 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.605197 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:46.605214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:46.606807 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:46.608732 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.608743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.608749 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.608989 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.609294 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.609297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.609299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.609386 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.609836 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:46.609842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:46.609978 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 16:36:46.610120 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 16:36:46.610350 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":116,"threads_started":1}
I20260927 16:36:46.610538 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.599654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.0.98.1:34425)
0927 16:36:46.600013 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.600024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.600426 (+ 402us) server_negotiation.cc:407] Connection header received
0927 16:36:46.600598 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.600608 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.600755 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.600893 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.601568 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.602798 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.604049 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.604453 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.604535 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.606951 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.606960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.606972 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.607002 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.608611 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.609118 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.609123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.609124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.609197 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.609478 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.609483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.609485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.609720 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.609864 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.610224 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.610388 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260927 16:36:46.611642 16443 tablet_service.cc:1514] Processing DeleteTablet for tablet 50139fc633d142cfb4cb880a576dc758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43238
I20260927 16:36:46.612679 16518 tablet_replica.cc:330] stopping tablet replica
I20260927 16:36:46.613030 16518 raft_consensus.cc:2229] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:46.613291 16518 raft_consensus.cc:2258] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:46.614292 16518 ts_tablet_manager.cc:1905] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:46.617679 16518 ts_tablet_manager.cc:1918] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:46.617786 16518 log.cc:1196] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal/wals/50139fc633d142cfb4cb880a576dc758
I20260927 16:36:46.618063 16518 ts_tablet_manager.cc:1939] T 50139fc633d142cfb4cb880a576dc758 P 8a47eba3c5b0481781e66cc47b3577c5: Deleting consensus metadata
I20260927 16:36:46.619678 16305 catalog_manager.cc:4841] TS 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425): tablet 50139fc633d142cfb4cb880a576dc758 (table test-table [id=d1e33786878c42ebbd55b8e8eb36e57b]) successfully deleted
W20260927 16:36:46.699185 16514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:46.699460 16514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:46.699530 16514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:46.703637 16514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:46.703698 16514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:46.703809 16514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:36:46.709233 16514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:46.710333 16514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:46.711335 16514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:46.721190 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.721215 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.721194 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.721294 16514 server_base.cc:1034] running on GCE node
I20260927 16:36:46.721823 16514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:46.722532 16514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:46.723686 16514 hybrid_clock.cc:648] HybridClock initialized: now 1790527006723660 us; error 39 us; skew 500 ppm
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:46.726502 16514 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:36:46.727342 16514 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260927 16:36:46.727928 16514 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:46.728006 16514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:46.732096 16514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 16:36:46.734747 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.736778 16514 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:36:46.736915 16514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "47634c73ac79473b856809590bac8bf8"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "3937ef42fe56e0e5f2c19417180e9dbd"
server_key_iv: "5324eca96274cefa6b6b0fd6b8b03108"
server_key_version: "encryptionkey@0"
I20260927 16:36:46.737802 16514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:46.762351 16514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:46.767361 16514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:46.767602 16514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:46.768574 16514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:46.769888 16514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:46.769965 16514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.770031 16514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:46.770077 16514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.783638 16514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45661
I20260927 16:36:46.783654 16646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45661 every 8 connection(s)
I20260927 16:36:46.784864 16514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:36:46.789412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16514
I20260927 16:36:46.789832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40837
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=44185
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:46.799660 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.786902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50025 (local address 127.0.98.62:43605)
0927 16:36:46.787040 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.787049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.788049 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 16:36:46.788970 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.788973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.789037 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.789138 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.790536 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.791077 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.792308 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.792559 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.792645 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.795837 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.795840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.795843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.795870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.797374 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.797958 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.797962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.797964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.798050 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.798481 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.798486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.798488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.798683 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.798798 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.799326 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.799479 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:36:46.800501 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.787220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:50025)
0927 16:36:46.787896 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:46.787937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:46.788740 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:46.789322 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:46.789328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:46.789749 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:46.790382 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.790400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.791205 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.791209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.792128 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.792161 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.792723 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.792730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.792992 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.793647 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:46.793670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:46.795635 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:46.797541 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.797548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.797565 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.797840 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.798145 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.798150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.798152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.798376 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.798797 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:46.798804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:46.799076 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 16:36:46.799270 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 16:36:46.799625 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":110,"threads_started":1}
I20260927 16:36:46.801812 16647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:46.802078 16647 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:46.802626 16647 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
I20260927 16:36:46.803787 16321 ts_manager.cc:194] Registered new tserver with Master: 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661)
I20260927 16:36:46.804378 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50025
W20260927 16:36:46.902532 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:36:46.902815 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:36:46.902890 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:36:46.907020 16650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:36:46.907095 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:36:46.907183 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:36:46.912567 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44185
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:36:46.913573 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:36:46.914638 16650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:36:46.923961 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:36:46.924003 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.924137 16650 server_base.cc:1034] running on GCE node
W20260927 16:36:46.923961 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:36:46.924751 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:36:46.925403 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:36:46.926581 16650 hybrid_clock.cc:648] HybridClock initialized: now 1790527006926541 us; error 58 us; skew 500 ppm
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:36:46.929450 16650 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:36:46.930395 16650 webserver.cc:460] Webserver started at http://127.0.98.3:44185/ using document root <none> and password file <none>
I20260927 16:36:46.930926 16650 fs_manager.cc:359] Metadata directory not provided
I20260927 16:36:46.930996 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:36:46.935148 16650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:36:46.937274 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.938669 16650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:36:46.938766 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "a0e5257fa2964130a34c6a90351df14e"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "944c625445d08a97f887f3879902b246"
server_key_iv: "cda383829bfd076d91a01cd5550b463e"
server_key_version: "encryptionkey@0"
I20260927 16:36:46.939275 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:36:46.951670 16650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:36:46.954893 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:36:46.955041 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:36:46.955564 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:36:46.956310 16650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:36:46.956354 16650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.956401 16650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:36:46.956418 16650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:36:46.965134 16650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40837
I20260927 16:36:46.965161 16778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40837 every 8 connection(s)
I20260927 16:36:46.966152 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:36:46 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:36:46.976626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16650
I20260927 16:36:46.980836 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.968302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37187 (local address 127.0.98.62:43605)
0927 16:36:46.968454 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.968458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.969613 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 16:36:46.970630 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.970633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.970686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.970774 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.971994 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.972525 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.974278 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.974512 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.974630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.977153 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.977156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.977158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.977204 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.978827 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.979366 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.979370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.979372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.979460 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.979802 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.979806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.979808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.979980 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.980088 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.980567 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.980679 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:36:46.981791 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.968744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:37187)
0927 16:36:46.969454 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:46.969495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:46.970415 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:46.970971 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:46.970979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:46.971356 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:36:46.971847 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.971861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.972670 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.972673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.974132 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:46.974141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.974667 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.974671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:46.974854 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.975432 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:36:46.975447 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:36:46.976960 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:36:46.978971 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.978977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.978992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.979251 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.979561 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:36:46.979564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:36:46.979566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:36:46.979706 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:36:46.980086 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:36:46.980093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:36:46.980336 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 16:36:46.980525 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 16:36:46.980847 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":120,"threads_started":1}
I20260927 16:36:46.983043 16779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43605
I20260927 16:36:46.983296 16779 heartbeater.cc:464] Registering TS with master...
I20260927 16:36:46.983815 16779 heartbeater.cc:511] Master 127.0.98.62:43605 requested a full tablet report, sending...
I20260927 16:36:46.985003 16321 ts_manager.cc:194] Registered new tserver with Master: a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
I20260927 16:36:46.985256 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.977355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.62:43605)
0927 16:36:46.977657 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.977661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.977760 (+ 99us) server_negotiation.cc:407] Connection header received
0927 16:36:46.977952 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.977955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.978006 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.978098 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.978852 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.979396 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.980256 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.980501 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.980609 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.982040 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.982043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.982045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.982065 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.983656 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.984096 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.984099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.984100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.984172 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.984463 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.984466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.984468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.984688 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.984895 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.985008 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.985122 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260927 16:36:46.985715 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37187
I20260927 16:36:46.986945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:36:46.990746 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.987574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.62:43605)
0927 16:36:46.987700 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.987704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.987865 (+ 161us) server_negotiation.cc:407] Connection header received
0927 16:36:46.987977 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.987979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.988055 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.988132 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:46.988420 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.988881 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.989732 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.989950 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.990024 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.990368 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:46.990483 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 16:36:46.990666 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46956: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 16:36:46.990818 16783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46956: token expired
W20260927 16:36:46.991225 16250 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43605)
I20260927 16:36:46.998123 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:46.991599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.0.98.62:43605)
0927 16:36:46.991705 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:46.991708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:46.991719 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:36:46.991826 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:46.991828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:46.991859 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:46.991933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:36:46.992535 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.992990 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.993795 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:46.994000 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:46.994074 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:46.995426 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:36:46.995430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:36:46.995432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:36:46.995452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:36:46.996884 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.997241 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.997244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.997245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.997324 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:36:46.997547 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:36:46.997549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:36:46.997550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:36:46.997694 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:36:46.997788 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:46.997945 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 16:36:46.998024 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:36:47.000546 16321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:36:47.004060 16321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:36:47.012670 16443 tablet_service.cc:1467] Processing CreateTablet for tablet d26328a62fc14586afdff372058de17d (DEFAULT_TABLE table=test-table [id=11b0c37ff51045a6bc6159c9ec9934a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:47.013417 16443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26328a62fc14586afdff372058de17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:47.016013 16401 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap starting.
I20260927 16:36:47.016894 16401 tablet_bootstrap.cc:654] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:47.017506 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.011607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40837 (local address 127.0.0.1:53836)
0927 16:36:47.011867 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:47.011911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:47.011999 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:47.012987 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:47.012989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:47.013007 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:47.013127 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.013135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.014772 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.014775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.015880 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.015888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.016975 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.016979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.017078 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.017129 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 16:36:47.017220 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 16:36:47.017336 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":2,"thread_start_us":127,"threads_started":1}
I20260927 16:36:47.017652 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.011378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45661 (local address 127.0.0.1:48268)
0927 16:36:47.011665 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:47.011681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:47.011774 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:47.012883 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:47.012886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:47.012904 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:47.013033 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.013040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.014430 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.014436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.015669 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.015679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.017227 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.017230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.017355 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.017388 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:36:47.017439 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 16:36:47.017512 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260927 16:36:47.018031 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.012019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.3:40837)
0927 16:36:47.012393 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:47.012406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:47.012451 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:36:47.012529 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:47.012534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:47.012753 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:47.013301 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:47.013580 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.014646 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.016006 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.016832 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.016961 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.017110 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:47.017343 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:36:47.017453 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":123,"threads_started":1}
I20260927 16:36:47.018316 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.011576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.98.2:45661)
0927 16:36:47.012465 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:47.012477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:47.012506 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:36:47.012569 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:47.012573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:47.012733 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:47.013015 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:47.013199 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.014315 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.015805 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.017093 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.017837 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.017979 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:47.018030 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:36:47.018131 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":147,"threads_started":1}
I20260927 16:36:47.018496 16401 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: No bootstrap required, opened a new log
I20260927 16:36:47.018579 16401 ts_tablet_manager.cc:1397] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:36:47.018981 16401 raft_consensus.cc:348] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.019109 16401 raft_consensus.cc:374] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:47.019155 16401 raft_consensus.cc:729] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a47eba3c5b0481781e66cc47b3577c5, State: Initialized, Role: FOLLOWER
I20260927 16:36:47.019328 16401 consensus_queue.cc:260] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.020200 16401 ts_tablet_manager.cc:1428] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:36:47.021157 16581 tablet_service.cc:1467] Processing CreateTablet for tablet d26328a62fc14586afdff372058de17d (DEFAULT_TABLE table=test-table [id=11b0c37ff51045a6bc6159c9ec9934a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:47.021123 16713 tablet_service.cc:1467] Processing CreateTablet for tablet d26328a62fc14586afdff372058de17d (DEFAULT_TABLE table=test-table [id=11b0c37ff51045a6bc6159c9ec9934a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:36:47.022125 16713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26328a62fc14586afdff372058de17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:47.022228 16581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26328a62fc14586afdff372058de17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:36:47.028740 16792 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Bootstrap starting.
I20260927 16:36:47.029042 16793 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Bootstrap starting.
I20260927 16:36:47.030985 16793 tablet_bootstrap.cc:654] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:47.031211 16792 tablet_bootstrap.cc:654] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Neither blocks nor log segments found. Creating new log.
I20260927 16:36:47.031910 16792 log.cc:824] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:47.032058 16793 log.cc:824] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:36:47.033828 16792 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: No bootstrap required, opened a new log
I20260927 16:36:47.033921 16793 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: No bootstrap required, opened a new log
I20260927 16:36:47.034104 16793 ts_tablet_manager.cc:1397] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:36:47.034294 16792 ts_tablet_manager.cc:1397] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 16:36:47.037101 16792 raft_consensus.cc:348] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.037156 16793 raft_consensus.cc:348] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.037398 16793 raft_consensus.cc:374] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:47.037405 16792 raft_consensus.cc:374] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:36:47.037462 16792 raft_consensus.cc:729] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47634c73ac79473b856809590bac8bf8, State: Initialized, Role: FOLLOWER
I20260927 16:36:47.037460 16793 raft_consensus.cc:729] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e5257fa2964130a34c6a90351df14e, State: Initialized, Role: FOLLOWER
I20260927 16:36:47.037869 16792 consensus_queue.cc:260] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.037881 16793 consensus_queue.cc:260] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.038537 16779 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
I20260927 16:36:47.038535 16647 heartbeater.cc:503] Master 127.0.98.62:43605 was elected leader, sending a full tablet report...
I20260927 16:36:47.038853 16792 ts_tablet_manager.cc:1428] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260927 16:36:47.039831 16793 ts_tablet_manager.cc:1428] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 16:36:47.072973 16510 tablet.cc:2369] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:36:47.097061 16796 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:36:47.097277 16796 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.098415 16796 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837), 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:47.102273 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.098702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40837 (local address 127.0.98.2:52899)
0927 16:36:47.098845 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:47.098863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:47.099011 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:47.099485 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:47.099488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:47.099505 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:47.099624 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.099629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.100352 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.100357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.101144 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.101152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.101806 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.101809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.101936 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.101954 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:36:47.102043 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:36:47.102115 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:36:47.102272 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.098800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52899 (local address 127.0.98.3:40837)
0927 16:36:47.098969 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:47.098974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:47.098990 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:36:47.099301 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:47.099307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:47.099368 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:47.099472 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:47.099732 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.100236 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.101266 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.101685 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.101808 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.101873 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:47.102059 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 16:36:47.102139 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:36:47.102895 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.099021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.98.2:54239)
0927 16:36:47.099397 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:47.099410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:36:47.099557 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:36:47.099840 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:36:47.099843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:36:47.099858 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:36:47.099989 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.099995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.100834 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.100837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.101602 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:36:47.101609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.102543 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.102546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:36:47.102639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.102672 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:36:47.102743 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:36:47.102792 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"spinlock_wait_cycles":768,"thread_start_us":145,"threads_started":1}
I20260927 16:36:47.102986 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.099173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54239 (local address 127.0.98.1:34425)
0927 16:36:47.099367 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:47.099372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:47.099510 (+ 138us) server_negotiation.cc:407] Connection header received
0927 16:36:47.099658 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:47.099661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:47.099724 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:47.099831 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:36:47.100104 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.100669 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.101727 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.102288 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.102418 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.102594 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:47.102790 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 16:36:47.102867 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 16:36:47.102981 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26328a62fc14586afdff372058de17d" candidate_uuid: "47634c73ac79473b856809590bac8bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e5257fa2964130a34c6a90351df14e" is_pre_election: true
I20260927 16:36:47.103336 16733 raft_consensus.cc:2454] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47634c73ac79473b856809590bac8bf8 in term 0.
I20260927 16:36:47.103539 16463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26328a62fc14586afdff372058de17d" candidate_uuid: "47634c73ac79473b856809590bac8bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47eba3c5b0481781e66cc47b3577c5" is_pre_election: true
I20260927 16:36:47.103840 16463 raft_consensus.cc:2454] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47634c73ac79473b856809590bac8bf8 in term 0.
I20260927 16:36:47.103873 16534 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e; no voters:
I20260927 16:36:47.104151 16796 raft_consensus.cc:2781] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:36:47.104223 16796 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:36:47.104285 16796 raft_consensus.cc:3037] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:47.105320 16796 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.105670 16796 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [CANDIDATE]: Term 1 election: Requested vote from peers a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837), 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425)
I20260927 16:36:47.106102 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26328a62fc14586afdff372058de17d" candidate_uuid: "47634c73ac79473b856809590bac8bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e5257fa2964130a34c6a90351df14e"
I20260927 16:36:47.106143 16463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26328a62fc14586afdff372058de17d" candidate_uuid: "47634c73ac79473b856809590bac8bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47eba3c5b0481781e66cc47b3577c5"
I20260927 16:36:47.106257 16733 raft_consensus.cc:3037] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:47.106317 16463 raft_consensus.cc:3037] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:36:47.107554 16463 raft_consensus.cc:2454] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47634c73ac79473b856809590bac8bf8 in term 1.
I20260927 16:36:47.107554 16733 raft_consensus.cc:2454] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47634c73ac79473b856809590bac8bf8 in term 1.
I20260927 16:36:47.107839 16534 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47634c73ac79473b856809590bac8bf8, 8a47eba3c5b0481781e66cc47b3577c5; no voters:
I20260927 16:36:47.108117 16796 raft_consensus.cc:2781] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:36:47.108403 16796 raft_consensus.cc:686] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 1 LEADER]: Becoming Leader. State: Replica: 47634c73ac79473b856809590bac8bf8, State: Running, Role: LEADER
I20260927 16:36:47.108731 16796 consensus_queue.cc:237] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:36:47.111920 16320 catalog_manager.cc:5495] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47634c73ac79473b856809590bac8bf8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "47634c73ac79473b856809590bac8bf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260927 16:36:47.140825 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:36:47.137192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.98.2:45661)
0927 16:36:47.137349 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:36:47.137354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:36:47.137368 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:36:47.137532 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:36:47.137534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:36:47.137590 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:36:47.137725 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:36:47.137975 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.138551 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.139388 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:36:47.139611 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:36:47.139720 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:36:47.140027 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:36:47.140446 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:36:47.140583 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:36:47.140624 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 16:36:47.140677 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:36:47.146859 16733 raft_consensus.cc:1261] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Refusing update from remote peer 47634c73ac79473b856809590bac8bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:47.146855 16463 raft_consensus.cc:1261] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Refusing update from remote peer 47634c73ac79473b856809590bac8bf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:36:47.147538 16796 consensus_queue.cc:1035] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:47.147712 16799 consensus_queue.cc:1035] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:36:47.154742 16804 mvcc.cc:205] Tried to move back new op lower bound from 7333998621266341888 to 7333998621119836160. Current Snapshot: MvccSnapshot[applied={T|T < 7333998621266341888}]
I20260927 16:36:47.162708 16321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:47.162878 16321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:36:47.165094 16321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Sending DeleteTablet for 3 replicas of tablet d26328a62fc14586afdff372058de17d
I20260927 16:36:47.165740 16581 tablet_service.cc:1514] Processing DeleteTablet for tablet d26328a62fc14586afdff372058de17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48268
I20260927 16:36:47.165789 16443 tablet_service.cc:1514] Processing DeleteTablet for tablet d26328a62fc14586afdff372058de17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43238
I20260927 16:36:47.165833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16359
I20260927 16:36:47.165915 16713 tablet_service.cc:1514] Processing DeleteTablet for tablet d26328a62fc14586afdff372058de17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53836
I20260927 16:36:47.166641 16812 tablet_replica.cc:330] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: stopping tablet replica
I20260927 16:36:47.166962 16812 raft_consensus.cc:2229] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:36:47.167362 16812 raft_consensus.cc:2258] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:47.170711 16813 tablet_replica.cc:330] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: stopping tablet replica
I20260927 16:36:47.171054 16813 raft_consensus.cc:2229] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:36:47.171084 16812 ts_tablet_manager.cc:1905] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:36:47.171378 16813 raft_consensus.cc:2258] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:36:47.173331 16812 ts_tablet_manager.cc:1918] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:47.173412 16812 log.cc:1196] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/d26328a62fc14586afdff372058de17d
I20260927 16:36:47.173660 16812 ts_tablet_manager.cc:1939] T d26328a62fc14586afdff372058de17d P 47634c73ac79473b856809590bac8bf8: Deleting consensus metadata
I20260927 16:36:47.174866 16305 catalog_manager.cc:4841] TS 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): tablet d26328a62fc14586afdff372058de17d (table test-table [id=11b0c37ff51045a6bc6159c9ec9934a1]) successfully deleted
W20260927 16:36:47.175697 16305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34425 (error 108)
I20260927 16:36:47.176136 16813 ts_tablet_manager.cc:1905] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:36:47.176941 16516 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:36:47.176635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34425 (local address 127.0.0.1:43240)
0927 16:36:47.176786 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:36:47.176854 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34425: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 16:36:47.177073 16305 catalog_manager.cc:4584] TS 8a47eba3c5b0481781e66cc47b3577c5 (127.0.98.1:34425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d26328a62fc14586afdff372058de17d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34425: connect: Connection reset by peer (error 104)
I20260927 16:36:47.177906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16514
I20260927 16:36:47.179244 16813 ts_tablet_manager.cc:1918] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:36:47.179359 16813 log.cc:1196] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/d26328a62fc14586afdff372058de17d
I20260927 16:36:47.179672 16813 ts_tablet_manager.cc:1939] T d26328a62fc14586afdff372058de17d P a0e5257fa2964130a34c6a90351df14e: Deleting consensus metadata
I20260927 16:36:47.180774 16305 catalog_manager.cc:4841] TS a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): tablet d26328a62fc14586afdff372058de17d (table test-table [id=11b0c37ff51045a6bc6159c9ec9934a1]) successfully deleted
I20260927 16:36:47.183754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16650
I20260927 16:36:47.189124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16288
I20260927 16:37:07.195936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--webserver_interface=127.0.98.62
--webserver_port=42407
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:07.308363 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:07.308640 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:07.308681 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:07.312836 16815 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:37:07.312896 16815 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:37:07.312912 16815 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:37:07.312927 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:07.312944 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:07.312963 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:07.313020 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:07.313051 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:07.318320 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:07.319353 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:07.320463 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:07.329386 16815 server_base.cc:1034] running on GCE node
W20260927 16:37:07.329444 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:07.329461 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:07.329686 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.330421 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:07.331527 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:07.332690 16815 hybrid_clock.cc:648] HybridClock initialized: now 1790527027332678 us; error 28 us; skew 500 ppm
Sep 27 16:37:07 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:07.335707 16815 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:07.336621 16815 webserver.cc:460] Webserver started at http://127.0.98.62:42407/ using document root <none> and password file <none>
I20260927 16:37:07.337214 16815 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:07.337280 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:07.341252 16815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:07.343490 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.344787 16815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:07.344928 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "8d06a49b60ef44cabdb2887c27cce01f"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "fdcc27ba46353023997737ceb0d67839"
server_key_iv: "10e45e89777c93d5dcf4af4b3d841827"
server_key_version: "encryptionkey@0"
I20260927 16:37:07.345471 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:07.368832 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:07.371850 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:07.372052 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:07.379390 16815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43605
I20260927 16:37:07.379705 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43605 every 8 connection(s)
I20260927 16:37:07.380568 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:07.382669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16815
I20260927 16:37:07.383165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34425
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=46471
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:37:07.385617 16883 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:37:07.387337 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap starting.
I20260927 16:37:07.404484 16883 log.cc:824] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:07.415431 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:37:07.415844 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Bootstrap complete.
I20260927 16:37:07.420049 16883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:37:07.420431 16883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Initialized, Role: FOLLOWER
I20260927 16:37:07.420917 16883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:37:07.421041 16883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:07.421139 16883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:07.421260 16883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:37:07.423287 16883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:37:07.423631 16883 leader_election.cc:302] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d06a49b60ef44cabdb2887c27cce01f; no voters:
I20260927 16:37:07.423976 16883 leader_election.cc:288] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:37:07.424135 16887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:37:07.424402 16887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [term 3 LEADER]: Becoming Leader. State: Replica: 8d06a49b60ef44cabdb2887c27cce01f, State: Running, Role: LEADER
I20260927 16:37:07.424732 16887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } }
I20260927 16:37:07.425451 16883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:07.425827 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:37:07.425980 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:07.426748 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d06a49b60ef44cabdb2887c27cce01f. Latest consensus state: current_term: 3 leader_uuid: "8d06a49b60ef44cabdb2887c27cce01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06a49b60ef44cabdb2887c27cce01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43605 } } }
I20260927 16:37:07.426862 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:07.427392 16891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:07.432839 16891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:07.433490 16891 catalog_manager.cc:1259] Loaded cluster ID: 5bfb6f95c9174625bce43cdc50bcc502
I20260927 16:37:07.433563 16891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:07.436429 16891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:07.437218 16891 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Loaded TSK: 0
I20260927 16:37:07.437309 16891 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d06a49b60ef44cabdb2887c27cce01f: Loaded TSK: 1
F20260927 16:37:07.438102 16891 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f70fab269ad google::LogMessage::Fail() at ??:0
@ 0x7f70fab2988a google::LogMessage::SendToLog() at ??:0
@ 0x7f70fab264de google::LogMessage::Flush() at ??:0
@ 0x7f70fab27a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f70fd7ea18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f70fd7e2f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f710548b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f710546599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7105466178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f71054a7df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f71054c682b std::function<>::operator()() at ??:0
@ 0x7f7105465b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7105466f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7105463579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f71054a7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f70fcd783d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f70fcd78c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f70fcd7a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f70fcd69512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70fe3ad6db start_thread at ??:0
@ 0x7f70f962461f clone at ??:0
W20260927 16:37:07.536280 16885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:07.536543 16885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:07.536588 16885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:07.540776 16885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:07.540838 16885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:07.540931 16885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:07.546586 16885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46471
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:07.547719 16885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:07.548812 16885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:07.559968 16931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:07.559931 16930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.560103 16885 server_base.cc:1034] running on GCE node
W20260927 16:37:07.560537 16933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.560936 16885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:07.561591 16885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:07.562773 16885 hybrid_clock.cc:648] HybridClock initialized: now 1790527027562738 us; error 56 us; skew 500 ppm
Sep 27 16:37:07 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:07.565713 16885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:37:07.566824 16885 webserver.cc:460] Webserver started at http://127.0.98.1:46471/ using document root <none> and password file <none>
I20260927 16:37:07.567545 16885 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:07.567636 16885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:07.572757 16885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:37:07.575485 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.577030 16885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 16:37:07.577160 16885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "8a47eba3c5b0481781e66cc47b3577c5"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "da707a725e11ea0d8224e74ca6d014ed"
server_key_iv: "9dda97470d83afe0d3173996e851e664"
server_key_version: "encryptionkey@0"
I20260927 16:37:07.577934 16885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:07.611949 16885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:07.615758 16885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:07.615972 16885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:07.616741 16885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:07.618163 16947 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:37:07.622648 16885 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:37:07.622732 16885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 16:37:07.622823 16885 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:37:07.625233 16885 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:37:07.625296 16885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260927 16:37:07.625587 16947 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap starting.
I20260927 16:37:07.636409 16885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34425
I20260927 16:37:07.636587 17054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34425 every 8 connection(s)
I20260927 16:37:07.637567 16885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:37:07.641216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 16885
I20260927 16:37:07.641696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45661
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:37:07.645630 16947 log.cc:824] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:07.652585 16947 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:37:07.652963 16947 tablet_bootstrap.cc:492] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Bootstrap complete.
I20260927 16:37:07.654261 16947 ts_tablet_manager.cc:1397] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.004s
I20260927 16:37:07.656790 16947 raft_consensus.cc:348] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:07.657239 16947 raft_consensus.cc:729] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a47eba3c5b0481781e66cc47b3577c5, State: Initialized, Role: FOLLOWER
I20260927 16:37:07.657775 16947 consensus_queue.cc:260] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:07.658844 16947 ts_tablet_manager.cc:1428] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 16:37:07.682750 17057 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:37:07.639889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:45235)
0927 16:37:07.640462 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.640522 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:07.641588 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:07.681880 (+ 40292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
W20260927 16:37:07.683405 17055 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43605 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43605: Client connection negotiation failed: client connection to 127.0.98.62:43605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:37:07.684234 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:07.683961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:47089)
0927 16:37:07.684104 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.684177 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 16:37:07.684974 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:07.684723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:43625)
0927 16:37:07.684847 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.684907 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 16:37:07.685191 17055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:37:07.766188 17058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:07.766433 17058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:07.766510 17058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:07.770524 17058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:07.770573 17058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:07.770674 17058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:07.775869 17058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:07.776950 17058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:07.778034 17058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:07.788055 17065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:07.788043 17068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.788224 17058 server_base.cc:1034] running on GCE node
W20260927 16:37:07.788406 17066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.788832 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:07.789486 17058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:07.790639 17058 hybrid_clock.cc:648] HybridClock initialized: now 1790527027790622 us; error 29 us; skew 500 ppm
Sep 27 16:37:07 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:07.793329 17058 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:37:07.794260 17058 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260927 16:37:07.794945 17058 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:07.795013 17058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:07.798974 17058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:07.801330 17075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.802675 17058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:37:07.802814 17058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "47634c73ac79473b856809590bac8bf8"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "3937ef42fe56e0e5f2c19417180e9dbd"
server_key_iv: "5324eca96274cefa6b6b0fd6b8b03108"
server_key_version: "encryptionkey@0"
I20260927 16:37:07.803391 17058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:07.815986 17058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:07.819005 17058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:07.819216 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:07.819828 17058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:07.820672 17058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:07.820741 17058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.820813 17058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:07.820870 17058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.830365 17058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45661
I20260927 16:37:07.830442 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45661 every 8 connection(s)
I20260927 16:37:07.831398 17058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:07.835210 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:07.833636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:46211)
0927 16:37:07.834392 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.834508 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":102,"threads_started":1}
W20260927 16:37:07.835707 17189 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43605 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43605: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
I20260927 16:37:07.836267 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:07.836071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:51535)
0927 16:37:07.836173 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.836216 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:37:07.836853 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:07.836675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:57253)
0927 16:37:07.836767 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:07.836805 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:37:07.837054 17189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:37:07.840768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17058
I20260927 16:37:07.841202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40837
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43605
--webserver_port=44185
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:07.953904 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:07.954285 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:07.954327 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:07.958474 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:07.958530 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:07.958590 17192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:07.963680 17192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44185
--tserver_master_addrs=127.0.98.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:07.964697 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:07.965668 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:07.975632 17197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.975701 17192 server_base.cc:1034] running on GCE node
W20260927 16:37:07.975642 17198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:07.975652 17200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:07.976434 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:07.977069 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:07.978245 17192 hybrid_clock.cc:648] HybridClock initialized: now 1790527027978207 us; error 50 us; skew 500 ppm
Sep 27 16:37:07 dist-test-slave-t27c krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:07.981084 17192 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:37:07.981977 17192 webserver.cc:460] Webserver started at http://127.0.98.3:44185/ using document root <none> and password file <none>
I20260927 16:37:07.982609 17192 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:07.982703 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:07.986707 17192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 16:37:07.988603 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:07.989785 17192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:07.989913 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "a0e5257fa2964130a34c6a90351df14e"
format_stamp: "Formatted at 2026-09-27 16:36:25 on dist-test-slave-t27c"
server_key: "944c625445d08a97f887f3879902b246"
server_key_iv: "cda383829bfd076d91a01cd5550b463e"
server_key_version: "encryptionkey@0"
I20260927 16:37:07.990569 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:08.005990 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:08.009431 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:08.009641 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:08.010258 17192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:08.011077 17192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:08.011147 17192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:08.011220 17192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:08.011263 17192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:08.020243 17192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40837
I20260927 16:37:08.020268 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40837 every 8 connection(s)
I20260927 16:37:08.021288 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:08.025203 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:08.023621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:37177)
0927 16:37:08.024299 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:08.024407 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":115,"threads_started":1}
W20260927 16:37:08.025686 17321 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43605 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43605: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
I20260927 16:37:08.026346 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:08.026116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:37167)
0927 16:37:08.026252 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:08.026295 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 16:37:08.026917 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:08.026745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:45191)
0927 16:37:08.026835 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:08.026874 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 16:37:08.027124 17321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:37:08.027348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17192
I20260927 16:37:08.686373 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:08.685835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:38877)
0927 16:37:08.686148 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:08.686266 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260927 16:37:08.838004 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:08.837569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:43441)
0927 16:37:08.837826 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:08.837900 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
I20260927 16:37:09.027977 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.027595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:57731)
0927 16:37:09.027825 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:09.027878 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260927 16:37:09.489370 17328 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:09.489567 17328 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:09.490752 17328 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
Sep 27 16:37:09 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 16:37:09 dist-test-slave-t27c krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 16:37:09.513463 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.491246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40837 (local address 127.0.98.1:46451)
0927 16:37:09.491518 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:09.491531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:09.491617 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:09.493029 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:09.493035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:09.493623 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:09.494517 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:09.494538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.495891 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.495893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:09.496772 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:09.496781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.497492 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.497495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:09.497721 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:09.506604 (+ 8883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:09.506632 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:09.508770 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:09.511194 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511196 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:09.511198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:09.511450 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:09.511844 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:09.511849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:09.511938 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:09.512550 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:09.512552 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:09.512710 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 16:37:09.512899 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 16:37:09.513267 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":217,"spinlock_wait_cycles":43904,"thread_start_us":49,"threads_started":1}
I20260927 16:37:09.513473 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.491104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50085 (local address 127.0.98.2:45661)
0927 16:37:09.491575 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:09.491594 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:09.491642 (+ 48us) server_negotiation.cc:407] Connection header received
0927 16:37:09.492334 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:09.492339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:09.492684 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:09.492972 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:09.494712 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.495749 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.496930 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.497368 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.497538 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:09.508778 (+ 11240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:09.508786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:09.508807 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:09.508852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:09.510675 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511291 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:09.511300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:09.511307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:09.511462 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511844 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:09.511847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:09.511849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:09.512351 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:09.512454 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:09.513007 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 16:37:09.513296 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":95,"threads_started":1}
I20260927 16:37:09.513473 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.491431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46451 (local address 127.0.98.3:40837)
0927 16:37:09.491702 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:09.491722 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:09.491755 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:37:09.492351 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:09.492356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:09.492696 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:09.492985 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:09.494721 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.495758 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.496888 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.497368 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.497539 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:09.509004 (+ 11465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:09.509009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:09.509023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:09.509054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:09.511074 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511590 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:09.511595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:09.511602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:09.511739 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:09.512028 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:09.512030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:09.512031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:09.512427 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:09.512558 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:09.513008 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 16:37:09.513280 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260927 16:37:09.513521 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.490932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45661 (local address 127.0.98.1:50085)
0927 16:37:09.491169 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:09.491188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:09.491306 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:09.493030 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:09.493036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:09.493623 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:09.494517 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:09.494538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.495893 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.495897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:09.496779 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:09.496788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:09.497498 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:09.497502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:09.497727 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:09.506604 (+ 8877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:09.506630 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:09.508435 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:09.510806 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:09.510815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:09.510831 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:09.511177 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:09.511567 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:09.511570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:09.511572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:09.511739 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:09.512473 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:09.512483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:09.512712 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 16:37:09.512899 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 16:37:09.513350 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":454,"spinlock_wait_cycles":12288,"thread_start_us":66,"threads_started":1}
W20260927 16:37:09.516239 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
W20260927 16:37:09.516501 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
I20260927 16:37:09.516600 16941 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5; no voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e
I20260927 16:37:09.516876 17328 raft_consensus.cc:2726] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:37:09.687407 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.687122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:60359)
0927 16:37:09.687253 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:09.687321 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:37:09.839139 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:09.838724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:39019)
0927 16:37:09.838969 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:09.839026 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260927 16:37:10.029248 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:10.028765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:41279)
0927 16:37:10.029069 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:10.029139 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260927 16:37:10.688544 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:10.688141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:51295)
0927 16:37:10.688387 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:10.688442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260927 16:37:10.840399 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:10.839931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:34427)
0927 16:37:10.840211 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:10.840288 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260927 16:37:11.030476 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:11.029988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:40431)
0927 16:37:11.030307 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:11.030365 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260927 16:37:11.689817 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:11.689343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:36507)
0927 16:37:11.689650 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:11.689714 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
I20260927 16:37:11.713610 17339 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:11.713766 17339 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:11.714200 17339 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
W20260927 16:37:11.715078 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
W20260927 16:37:11.715260 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
I20260927 16:37:11.715319 16941 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5; no voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e
I20260927 16:37:11.715509 17339 raft_consensus.cc:2726] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:37:11.841692 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:11.841259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:58821)
0927 16:37:11.841536 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:11.841601 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260927 16:37:12.031636 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:12.031241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:45755)
0927 16:37:12.031485 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:12.031541 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260927 16:37:12.691390 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:12.690833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:43177)
0927 16:37:12.691228 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:12.691290 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":153,"threads_started":1}
I20260927 16:37:12.842945 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:12.842502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:51651)
0927 16:37:12.842784 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:12.842847 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260927 16:37:13.032737 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:13.032342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:45825)
0927 16:37:13.032595 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:13.032647 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 16:37:13.692610 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:13.692182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:36189)
0927 16:37:13.692428 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:13.692486 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260927 16:37:13.809684 17346 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:13.809814 17346 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:13.810253 17346 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
W20260927 16:37:13.811043 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
W20260927 16:37:13.811210 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
I20260927 16:37:13.811290 16941 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5; no voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e
I20260927 16:37:13.811456 17346 raft_consensus.cc:2726] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:37:13.844334 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:13.843746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:38795)
0927 16:37:13.844172 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:13.844235 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":73,"threads_started":1}
I20260927 16:37:14.033924 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:14.033493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:39779)
0927 16:37:14.033769 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:14.033831 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260927 16:37:14.693958 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:14.693503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:37017)
0927 16:37:14.693796 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:14.693856 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260927 16:37:14.845767 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:14.845237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:42125)
0927 16:37:14.845584 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:14.845647 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260927 16:37:15.035271 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:15.034803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:43801)
0927 16:37:15.035102 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:15.035164 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260927 16:37:15.695230 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:15.694823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:42769)
0927 16:37:15.695080 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:15.695133 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260927 16:37:15.824550 17353 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:15.824712 17353 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:15.825199 17353 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
W20260927 16:37:15.825989 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
W20260927 16:37:15.826155 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
I20260927 16:37:15.826241 16941 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5; no voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e
I20260927 16:37:15.826398 17353 raft_consensus.cc:2726] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:37:15.846982 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:15.846594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:59313)
0927 16:37:15.846835 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:15.846889 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260927 16:37:16.036482 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:16.036059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:47205)
0927 16:37:16.036330 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:16.036388 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260927 16:37:16.696619 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:16.696171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:42901)
0927 16:37:16.696442 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:16.696510 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260927 16:37:16.848067 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:16.847673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:47877)
0927 16:37:16.847913 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:16.847971 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260927 16:37:17.037742 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:17.037359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:55383)
0927 16:37:17.037597 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:17.037650 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260927 16:37:17.697868 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:17.697492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:47067)
0927 16:37:17.697720 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:17.697772 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260927 16:37:17.849524 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:17.849048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:42327)
0927 16:37:17.849310 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:17.849365 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260927 16:37:18.039081 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:18.038672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:54557)
0927 16:37:18.038927 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:18.038984 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260927 16:37:18.699260 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:18.698835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:50207)
0927 16:37:18.699096 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:18.699158 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260927 16:37:18.850733 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:18.850315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:54591)
0927 16:37:18.850581 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:18.850636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260927 16:37:19.040355 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:19.039873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:45551)
0927 16:37:19.040194 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:19.040258 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260927 16:37:19.700439 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:19.700016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:60665)
0927 16:37:19.700281 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:19.700338 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260927 16:37:19.851853 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:19.851471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:48947)
0927 16:37:19.851704 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:19.851758 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260927 16:37:20.041635 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:20.041174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:55925)
0927 16:37:20.041471 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:20.041531 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
I20260927 16:37:20.701610 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:20.701198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:43563)
0927 16:37:20.701450 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:20.701507 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260927 16:37:20.853106 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:20.852615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:40543)
0927 16:37:20.852939 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:20.853003 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
I20260927 16:37:21.042903 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:21.042418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:49943)
0927 16:37:21.042743 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:21.042805 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260927 16:37:21.703054 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:21.702483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:40075)
0927 16:37:21.702809 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:21.702900 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260927 16:37:21.854470 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:21.853986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:50279)
0927 16:37:21.854317 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:21.854374 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260927 16:37:22.044083 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:22.043662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:60523)
0927 16:37:22.043931 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:22.043989 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260927 16:37:22.098397 17374 raft_consensus.cc:482] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:22.098567 17374 raft_consensus.cc:504] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47634c73ac79473b856809590bac8bf8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45661 } } peers { permanent_uuid: "a0e5257fa2964130a34c6a90351df14e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40837 } } peers { permanent_uuid: "8a47eba3c5b0481781e66cc47b3577c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34425 } }
I20260927 16:37:22.099006 17374 leader_election.cc:288] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661), a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837)
W20260927 16:37:22.099910 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0e5257fa2964130a34c6a90351df14e (127.0.98.3:40837): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
W20260927 16:37:22.100040 16941 leader_election.cc:341] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47634c73ac79473b856809590bac8bf8 (127.0.98.2:45661): Not found: Tablet not found: d26328a62fc14586afdff372058de17d
I20260927 16:37:22.100103 16941 leader_election.cc:302] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a47eba3c5b0481781e66cc47b3577c5; no voters: 47634c73ac79473b856809590bac8bf8, a0e5257fa2964130a34c6a90351df14e
I20260927 16:37:22.100291 17374 raft_consensus.cc:2726] T d26328a62fc14586afdff372058de17d P 8a47eba3c5b0481781e66cc47b3577c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:37:22.704491 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:22.704063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.1:52707)
0927 16:37:22.704332 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:22.704393 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260927 16:37:22.855742 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:22.855329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.2:59715)
0927 16:37:22.855594 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:22.855649 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260927 16:37:23.034459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 16885
I20260927 16:37:23.040333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17058
I20260927 16:37:23.045387 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:23.044979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43605 (local address 127.0.98.3:52355)
0927 16:37:23.045240 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:23.045295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260927 16:37:23.045883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17192
2026-09-27T16:37:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59225 ms)
[----------] 30 tests from SecurityITest (240489 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:23 dist-test-slave-t27c krb5kdc[17380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:23 dist-test-slave-t27c krb5kdc[17380](info): set up 2 sockets
Sep 27 16:37:23 dist-test-slave-t27c krb5kdc[17380](info): commencing operation
krb5kdc: starting...
W20260927 16:37:24.093866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:24.217042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:24.327978 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:24.328243 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:24.328284 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:24.332340 17393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:24.332393 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:24.332412 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:24.332429 17393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:24.332487 17393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:24.337704 17393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:24.338819 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:24.339864 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:24.349433 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:24.349488 17393 server_base.cc:1034] running on GCE node
W20260927 16:37:24.349469 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:24.349465 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:24.350427 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:24.351598 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:24.352797 17393 hybrid_clock.cc:648] HybridClock initialized: now 1790527044352760 us; error 48 us; skew 500 ppm
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:24.355665 17393 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:24.356545 17393 webserver.cc:460] Webserver started at http://127.0.98.62:38987/ using document root <none> and password file <none>
I20260927 16:37:24.357067 17393 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:24.357126 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:24.357292 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:24.359418 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "c3134031da2d4a1cb29b81623a2ba0ec"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "553c3f9ddb7c37f6cd2677b174af0ee4"
server_key_iv: "dffd27d907476ce636b1d5144966e49c"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.359952 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3134031da2d4a1cb29b81623a2ba0ec"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "553c3f9ddb7c37f6cd2677b174af0ee4"
server_key_iv: "dffd27d907476ce636b1d5144966e49c"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.363679 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:37:24.366153 17408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.367484 17393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:24.367585 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "c3134031da2d4a1cb29b81623a2ba0ec"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "553c3f9ddb7c37f6cd2677b174af0ee4"
server_key_iv: "dffd27d907476ce636b1d5144966e49c"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.367723 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:24.380416 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:24.383723 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:24.383937 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:24.391232 17393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40155
I20260927 16:37:24.391264 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40155 every 8 connection(s)
I20260927 16:37:24.392241 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:24.393186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17393
I20260927 16:37:24.393356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:24.393671 392 external_mini_cluster.cc:1428] Setting key 7f1615b7f1561ddce70c5d9b5e8524ce
I20260927 16:37:24.395720 17461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:24.401258 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: Bootstrap starting.
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:24.403643 17461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:24.404523 17461 log.cc:824] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:24.406831 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: No bootstrap required, opened a new log
I20260927 16:37:24.407989 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:24.395190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.98.62:40155)
0927 16:37:24.395660 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:24.395677 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:24.395715 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:37:24.396438 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:24.396455 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:24.396782 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:24.397131 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:24.397931 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.399015 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.399880 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.400350 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.400513 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:24.403075 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:24.403081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:24.403094 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:24.403120 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:24.405020 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.405529 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.405534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.405538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.405645 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.405927 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.405929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.405930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.406391 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:24.406541 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:24.406830 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 16:37:24.407253 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":165,"threads_started":1}
I20260927 16:37:24.410506 17461 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } }
I20260927 16:37:24.410727 17461 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:24.410760 17461 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3134031da2d4a1cb29b81623a2ba0ec, State: Initialized, Role: FOLLOWER
I20260927 16:37:24.411206 17461 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } }
I20260927 16:37:24.411348 17461 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:24.411412 17461 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:24.411535 17461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:24.412492 17461 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } }
I20260927 16:37:24.412833 17461 leader_election.cc:302] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3134031da2d4a1cb29b81623a2ba0ec; no voters:
I20260927 16:37:24.413127 17461 leader_election.cc:288] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:24.413269 17466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:24.413512 17466 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [term 1 LEADER]: Becoming Leader. State: Replica: c3134031da2d4a1cb29b81623a2ba0ec, State: Running, Role: LEADER
I20260927 16:37:24.413817 17466 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } }
I20260927 16:37:24.414381 17461 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:24.415558 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3134031da2d4a1cb29b81623a2ba0ec. Latest consensus state: current_term: 1 leader_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } } }
I20260927 16:37:24.415706 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:24.416028 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3134031da2d4a1cb29b81623a2ba0ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40155 } } }
I20260927 16:37:24.416121 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:24.416651 17475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:24.419452 17475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:24.424948 17475 catalog_manager.cc:1347] Generated new cluster ID: 16e5eb8277884eef901d34a4db3e204e
I20260927 16:37:24.425019 17475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:24.432652 17475 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:24.433959 17475 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:24.440111 17475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: Generated new TSK 0
I20260927 16:37:24.440919 17475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:37:24.472854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:24.584769 17487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:24.585050 17487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:24.585093 17487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:24.589213 17487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:24.589287 17487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:24.589386 17487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:24.594913 17487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:24.596042 17487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:24.597129 17487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:24.606808 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:24.606822 17493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:24.606822 17492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:24.606861 17487 server_base.cc:1034] running on GCE node
I20260927 16:37:24.607420 17487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:24.608126 17487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:24.609313 17487 hybrid_clock.cc:648] HybridClock initialized: now 1790527044609294 us; error 28 us; skew 500 ppm
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:24.612149 17487 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:37:24.613034 17487 webserver.cc:460] Webserver started at http://127.0.98.1:37455/ using document root <none> and password file <none>
I20260927 16:37:24.613631 17487 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:24.613720 17487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:24.613967 17487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:24.616257 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ee0415b32ce4acba9f48161e61c280f"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "77f8451793cfc31fa2215718f82ea499"
server_key_iv: "9e228d33c6e4318180217b1b4f88d663"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.616827 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ee0415b32ce4acba9f48161e61c280f"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "77f8451793cfc31fa2215718f82ea499"
server_key_iv: "9e228d33c6e4318180217b1b4f88d663"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.620561 17487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:37:24.622987 17502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.624130 17487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:24.624264 17487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "3ee0415b32ce4acba9f48161e61c280f"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "77f8451793cfc31fa2215718f82ea499"
server_key_iv: "9e228d33c6e4318180217b1b4f88d663"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.624387 17487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:24.635747 17487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:24.639261 17487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:24.639431 17487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:24.639971 17487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:24.640760 17487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:24.640808 17487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.640847 17487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:24.640870 17487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.649468 17487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42775
I20260927 16:37:24.649487 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42775 every 8 connection(s)
I20260927 16:37:24.650416 17487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:24.659650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17487
I20260927 16:37:24.659825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:24.660122 392 external_mini_cluster.cc:1428] Setting key 5dd26f3db9e5e935880b7d32d2048eb3
I20260927 16:37:24.664021 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:24.652226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52667 (local address 127.0.98.62:40155)
0927 16:37:24.652369 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:24.652373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:24.653207 (+ 834us) server_negotiation.cc:407] Connection header received
0927 16:37:24.654156 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:24.654159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:24.654245 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:24.654326 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:24.655500 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.656052 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.657213 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.657460 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.657576 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:24.660201 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:24.660206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:24.660208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:24.660237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:24.661919 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.662497 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.662502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.662504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.662587 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.662932 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.662937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.662939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.663123 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:24.663232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:24.663723 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 16:37:24.663861 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:37:24.664844 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:24.652605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40155 (local address 127.0.98.1:52667)
0927 16:37:24.653062 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:24.653105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:24.653924 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:24.654500 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:24.654507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:24.654859 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:24.655343 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:24.655357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.656187 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.656191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:24.657081 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:24.657090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.657614 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.657619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:24.657794 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:24.658432 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:24.658448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:24.660035 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:24.662038 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:24.662044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:24.662059 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:24.662395 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:24.662683 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:24.662688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:24.662690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:24.662835 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:24.663231 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:24.663238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:24.663491 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 16:37:24.663683 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 16:37:24.664014 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260927 16:37:24.666222 17616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40155
I20260927 16:37:24.666484 17616 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:24.666975 17616 heartbeater.cc:511] Master 127.0.98.62:40155 requested a full tablet report, sending...
I20260927 16:37:24.668586 17425 ts_manager.cc:194] Registered new tserver with Master: 3ee0415b32ce4acba9f48161e61c280f (127.0.98.1:42775)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:37:24.669864 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:37:24.687122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:24.796689 17621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:24.796974 17621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:24.797019 17621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:24.801054 17621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:24.801115 17621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:24.801234 17621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:24.806496 17621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:24.807570 17621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:24.808617 17621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:24.818334 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:24.818353 17621 server_base.cc:1034] running on GCE node
W20260927 16:37:24.818375 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:24.818377 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:24.818979 17621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:24.819656 17621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:24.820848 17621 hybrid_clock.cc:648] HybridClock initialized: now 1790527044820817 us; error 44 us; skew 500 ppm
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:24.823611 17621 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:37:24.824484 17621 webserver.cc:460] Webserver started at http://127.0.98.2:32969/ using document root <none> and password file <none>
I20260927 16:37:24.825067 17621 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:24.825161 17621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:24.825392 17621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:24.827659 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "caf4961fe2df4264aabda2b4f9debb31"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "45edc2f9375adfa8ebbdf753a32444ad"
server_key_iv: "66aafb118162dfd1dd7751d75a7dd686"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.828307 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "caf4961fe2df4264aabda2b4f9debb31"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "45edc2f9375adfa8ebbdf753a32444ad"
server_key_iv: "66aafb118162dfd1dd7751d75a7dd686"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.832231 17621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:37:24.834793 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.835989 17621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:24.836133 17621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "caf4961fe2df4264aabda2b4f9debb31"
format_stamp: "Formatted at 2026-09-27 16:37:24 on dist-test-slave-t27c"
server_key: "45edc2f9375adfa8ebbdf753a32444ad"
server_key_iv: "66aafb118162dfd1dd7751d75a7dd686"
server_key_version: "encryptionkey@0"
I20260927 16:37:24.836246 17621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:24.855361 17621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:24.858755 17621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:24.858975 17621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:24.859568 17621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:24.860410 17621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:24.860483 17621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.860548 17621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:24.860579 17621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:24.869506 17621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37813
I20260927 16:37:24.869508 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37813 every 8 connection(s)
I20260927 16:37:24.870584 17621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:24.873215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17621
I20260927 16:37:24.873355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:24.873703 392 external_mini_cluster.cc:1428] Setting key 6fc7e8d31d70f582c197dd79890e6e87
Sep 27 16:37:24 dist-test-slave-t27c krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:37:24.885202 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:24.872403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36489 (local address 127.0.98.62:40155)
0927 16:37:24.872553 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:24.872557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:24.873632 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 16:37:24.874781 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:24.874784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:24.874834 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:24.874905 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:24.876113 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.876952 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.878414 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.878632 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.878735 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:24.881600 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:24.881604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:24.881606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:24.881628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:24.883163 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.883661 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.883665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.883667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.883747 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:24.884098 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:24.884101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:24.884103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:24.884283 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:24.884390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:24.884886 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 16:37:24.885011 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:37:24.886165 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:24.872894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40155 (local address 127.0.98.2:36489)
0927 16:37:24.873427 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:24.873498 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:24.874561 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:24.875118 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:24.875125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:24.875489 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:24.875963 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:24.875976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.877079 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.877083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:24.878284 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:24.878298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:24.878804 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:24.878811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:24.879046 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:24.879693 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:24.879715 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:24.881430 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:24.883268 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:24.883274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:24.883287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:24.883553 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:24.883847 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:24.883850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:24.883852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:24.883994 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:24.884393 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:24.884402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:24.884648 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 16:37:24.884836 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 16:37:24.885224 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260927 16:37:24.887465 17750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40155
I20260927 16:37:24.887718 17750 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:24.888283 17750 heartbeater.cc:511] Master 127.0.98.62:40155 requested a full tablet report, sending...
I20260927 16:37:24.889355 17425 ts_manager.cc:194] Registered new tserver with Master: caf4961fe2df4264aabda2b4f9debb31 (127.0.98.2:37813)
I20260927 16:37:24.889956 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:37:24.901393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:25.014006 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:25.014349 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:25.014434 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:25.018704 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:25.018788 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:25.018873 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:25.024220 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:25.025305 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:25.026398 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:25.036567 17763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.036551 17760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.036618 17755 server_base.cc:1034] running on GCE node
W20260927 16:37:25.036616 17761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.037159 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:25.037792 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:25.038995 17755 hybrid_clock.cc:648] HybridClock initialized: now 1790527045038980 us; error 27 us; skew 500 ppm
Sep 27 16:37:25 dist-test-slave-t27c krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:25.041875 17755 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:37:25.042779 17755 webserver.cc:460] Webserver started at http://127.0.98.3:36429/ using document root <none> and password file <none>
I20260927 16:37:25.043349 17755 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:25.043442 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:25.043681 17755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:25.045692 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "baa18bc3f005394b7713d56797262e53"
server_key_iv: "0febd1f51edc9e210270dace6c3bc1ed"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.046326 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "baa18bc3f005394b7713d56797262e53"
server_key_iv: "0febd1f51edc9e210270dace6c3bc1ed"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.049901 17755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:37:25.052453 17770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.053704 17755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:25.053839 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "baa18bc3f005394b7713d56797262e53"
server_key_iv: "0febd1f51edc9e210270dace6c3bc1ed"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.053964 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:25.072273 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:25.075827 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:25.076056 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:25.076676 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:25.077517 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:25.077589 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.077663 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:25.077701 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.086604 17755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41679
I20260927 16:37:25.086635 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41679 every 8 connection(s)
I20260927 16:37:25.087527 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:37:25 dist-test-slave-t27c krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:25.097802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17755
I20260927 16:37:25.097947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:25.098279 392 external_mini_cluster.cc:1428] Setting key 908ba1e9da2f13615d39ff4dbd0c0479
I20260927 16:37:25.100666 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.089375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49843 (local address 127.0.98.62:40155)
0927 16:37:25.089522 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.089526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.090363 (+ 837us) server_negotiation.cc:407] Connection header received
0927 16:37:25.091227 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.091230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.091277 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.091355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:25.092601 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.093136 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.094259 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.094476 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.094556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.096847 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:25.096850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:25.096852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:25.096890 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:25.098560 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:25.099040 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:25.099043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:25.099046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:25.099137 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:25.099579 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:25.099581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:25.099583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:25.099811 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:25.099918 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.100436 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.100550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:37:25.101521 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.089762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40155 (local address 127.0.98.3:49843)
0927 16:37:25.090218 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.090261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.091033 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.091523 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.091530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.091990 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:25.092476 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.092489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.093243 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.093245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.094131 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.094139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.094603 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.094605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.094745 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.095291 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:25.095306 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:25.096688 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:25.098707 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:25.098711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:25.098723 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:25.098957 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:25.099254 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:25.099257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:25.099259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:25.099460 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:25.099922 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:25.099927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:25.100147 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.100367 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.100676 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260927 16:37:25.102568 17884 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40155
I20260927 16:37:25.102805 17884 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:25.103279 17884 heartbeater.cc:511] Master 127.0.98.62:40155 requested a full tablet report, sending...
I20260927 16:37:25.104269 17425 ts_manager.cc:194] Registered new tserver with Master: a92dc402bb764b9bbb8bb5a2c7b38cdf (127.0.98.3:41679)
I20260927 16:37:25.104852 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49843
I20260927 16:37:25.112257 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:25.122186 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.114349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.0.98.62:40155)
0927 16:37:25.114483 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.114489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.114694 (+ 205us) server_negotiation.cc:407] Connection header received
0927 16:37:25.114850 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.114853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.114906 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.114993 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:25.115750 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.116350 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.117213 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.117448 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.117559 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.118989 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:25.118993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:25.118996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:25.119025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:25.120445 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:25.120869 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:25.120874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:25.120878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:25.120969 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:25.121242 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:25.121246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:25.121248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:25.121507 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:25.121648 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.121889 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.122000 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:37:25.124883 17425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:25.128041 17425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:25.142313 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.136124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.98.2:37813)
0927 16:37:25.136917 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.136933 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.136963 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:37:25.137044 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.137051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.137242 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.137783 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:25.137962 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139185 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141080 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.141620 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141744 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.141905 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.142019 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.142110 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":86,"threads_started":1}
I20260927 16:37:25.142313 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.135992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37813 (local address 127.0.0.1:41448)
0927 16:37:25.136439 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.136486 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.136609 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.137439 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.137446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.137470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:25.137629 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.137636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139356 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.139360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.140861 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.140872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.141743 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.141850 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.141887 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.141994 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.142110 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":105,"thread_start_us":87,"threads_started":1}
I20260927 16:37:25.142352 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.136082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42775 (local address 127.0.0.1:53872)
0927 16:37:25.136843 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.136857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.136935 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.137619 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.137622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.137637 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:25.137767 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.137775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139375 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.139378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.140756 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.140767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.141891 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.141992 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.142014 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.142066 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.142132 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":68,"thread_start_us":107,"threads_started":1}
I20260927 16:37:25.143091 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.136812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.98.1:42775)
0927 16:37:25.137154 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.137167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.137189 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:37:25.137267 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.137276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.137485 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.137638 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:25.137901 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139185 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.140892 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.141747 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.142674 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.142803 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.142853 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.142942 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":76,"threads_started":1}
I20260927 16:37:25.143106 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.137163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.0.98.3:41679)
0927 16:37:25.137512 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.137523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.138048 (+ 525us) server_negotiation.cc:407] Connection header received
0927 16:37:25.138132 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.138136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.138322 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.138470 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:25.138725 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139649 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141060 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.141780 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.141889 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.142021 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.142882 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.142966 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260927 16:37:25.143514 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.136731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41679 (local address 127.0.0.1:42132)
0927 16:37:25.137907 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.137921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.138006 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.138462 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.138465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.138482 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:25.138588 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.138597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.139780 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.139783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.140951 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.140959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.142078 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.142081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.142205 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.142228 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.142292 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.142350 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":96,"threads_started":1}
I20260927 16:37:25.145195 17684 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 (DEFAULT_TABLE table=test-table [id=efc39faf4dd3492ca3041906627ad36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:25.145351 17550 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 (DEFAULT_TABLE table=test-table [id=efc39faf4dd3492ca3041906627ad36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:25.146349 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c03b8b9c4d4b9e825a9606287f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:25.146341 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c03b8b9c4d4b9e825a9606287f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:25.149098 17818 tablet_service.cc:1467] Processing CreateTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 (DEFAULT_TABLE table=test-table [id=efc39faf4dd3492ca3041906627ad36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:25.150316 17818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d1c03b8b9c4d4b9e825a9606287f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:25.153105 17901 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: Bootstrap starting.
I20260927 16:37:25.154584 17902 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: Bootstrap starting.
I20260927 16:37:25.156033 17901 tablet_bootstrap.cc:654] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:25.156822 17902 tablet_bootstrap.cc:654] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:25.156852 17903 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Bootstrap starting.
I20260927 16:37:25.156971 17901 log.cc:824] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:25.158069 17902 log.cc:824] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:25.159106 17903 tablet_bootstrap.cc:654] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:25.159916 17902 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: No bootstrap required, opened a new log
I20260927 16:37:25.160116 17902 ts_tablet_manager.cc:1397] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:37:25.160123 17901 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: No bootstrap required, opened a new log
I20260927 16:37:25.160210 17903 log.cc:824] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:25.160308 17901 ts_tablet_manager.cc:1397] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 16:37:25.162060 17903 tablet_bootstrap.cc:492] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: No bootstrap required, opened a new log
I20260927 16:37:25.162262 17903 ts_tablet_manager.cc:1397] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:37:25.163306 17902 raft_consensus.cc:348] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.163548 17902 raft_consensus.cc:374] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:25.163609 17902 raft_consensus.cc:729] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee0415b32ce4acba9f48161e61c280f, State: Initialized, Role: FOLLOWER
I20260927 16:37:25.164074 17902 consensus_queue.cc:260] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.164072 17901 raft_consensus.cc:348] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.164327 17901 raft_consensus.cc:374] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:25.164388 17901 raft_consensus.cc:729] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf4961fe2df4264aabda2b4f9debb31, State: Initialized, Role: FOLLOWER
I20260927 16:37:25.164806 17616 heartbeater.cc:503] Master 127.0.98.62:40155 was elected leader, sending a full tablet report...
I20260927 16:37:25.165086 17902 ts_tablet_manager.cc:1428] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:37:25.165822 17901 consensus_queue.cc:260] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.165939 17903 raft_consensus.cc:348] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.166216 17903 raft_consensus.cc:374] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:25.166276 17903 raft_consensus.cc:729] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92dc402bb764b9bbb8bb5a2c7b38cdf, State: Initialized, Role: FOLLOWER
I20260927 16:37:25.166693 17750 heartbeater.cc:503] Master 127.0.98.62:40155 was elected leader, sending a full tablet report...
I20260927 16:37:25.166715 17903 consensus_queue.cc:260] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.166844 17901 ts_tablet_manager.cc:1428] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:37:25.167482 17903 ts_tablet_manager.cc:1428] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 16:37:25.167848 17884 heartbeater.cc:503] Master 127.0.98.62:40155 was elected leader, sending a full tablet report...
I20260927 16:37:25.224963 17909 raft_consensus.cc:482] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:25.225190 17909 raft_consensus.cc:504] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.226454 17909 leader_election.cc:288] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caf4961fe2df4264aabda2b4f9debb31 (127.0.98.2:37813), 3ee0415b32ce4acba9f48161e61c280f (127.0.98.1:42775)
I20260927 16:37:25.230261 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.226665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37813 (local address 127.0.98.3:53837)
0927 16:37:25.226870 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.226887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.226986 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.227275 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.227277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.227294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:25.227431 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.227436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.228197 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.228202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.229115 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.229128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.229784 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.229788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.229907 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.229925 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.230012 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.230088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 16:37:25.230261 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.226844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53837 (local address 127.0.98.2:37813)
0927 16:37:25.226973 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.226977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.226994 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:37:25.227129 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.227131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.227172 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.227262 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:25.227537 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.228090 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.229249 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.229661 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.229779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.229845 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.230029 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.230100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:37:25.230549 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.226692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42775 (local address 127.0.98.3:46037)
0927 16:37:25.227052 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.227066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.227212 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.227520 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.227523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.227539 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:25.227661 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.227666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.228481 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.228484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.229386 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.229395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.230069 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.230072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.230225 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.230248 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.230339 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.230414 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
I20260927 16:37:25.230536 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.226812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46037 (local address 127.0.98.1:42775)
0927 16:37:25.227056 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.227063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.227159 (+ 96us) server_negotiation.cc:407] Connection header received
0927 16:37:25.227364 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.227373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.227416 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.227499 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:25.227770 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.228330 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.229509 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.229946 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.230065 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.230127 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.230357 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.230426 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 16:37:25.231105 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c03b8b9c4d4b9e825a9606287f20" candidate_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf4961fe2df4264aabda2b4f9debb31" is_pre_election: true
I20260927 16:37:25.231105 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c03b8b9c4d4b9e825a9606287f20" candidate_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee0415b32ce4acba9f48161e61c280f" is_pre_election: true
I20260927 16:37:25.231426 17704 raft_consensus.cc:2454] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a92dc402bb764b9bbb8bb5a2c7b38cdf in term 0.
I20260927 16:37:25.231408 17570 raft_consensus.cc:2454] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a92dc402bb764b9bbb8bb5a2c7b38cdf in term 0.
I20260927 16:37:25.231904 17772 leader_election.cc:302] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee0415b32ce4acba9f48161e61c280f, a92dc402bb764b9bbb8bb5a2c7b38cdf; no voters:
I20260927 16:37:25.232221 17909 raft_consensus.cc:2781] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:25.232306 17909 raft_consensus.cc:482] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:25.232349 17909 raft_consensus.cc:3037] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:25.233476 17909 raft_consensus.cc:504] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.233872 17909 leader_election.cc:288] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [CANDIDATE]: Term 1 election: Requested vote from peers caf4961fe2df4264aabda2b4f9debb31 (127.0.98.2:37813), 3ee0415b32ce4acba9f48161e61c280f (127.0.98.1:42775)
I20260927 16:37:25.234236 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c03b8b9c4d4b9e825a9606287f20" candidate_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee0415b32ce4acba9f48161e61c280f"
I20260927 16:37:25.234262 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d1c03b8b9c4d4b9e825a9606287f20" candidate_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf4961fe2df4264aabda2b4f9debb31"
I20260927 16:37:25.234349 17570 raft_consensus.cc:3037] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:25.234405 17704 raft_consensus.cc:3037] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:25.235412 17570 raft_consensus.cc:2454] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a92dc402bb764b9bbb8bb5a2c7b38cdf in term 1.
I20260927 16:37:25.235445 17704 raft_consensus.cc:2454] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a92dc402bb764b9bbb8bb5a2c7b38cdf in term 1.
I20260927 16:37:25.235770 17772 leader_election.cc:302] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee0415b32ce4acba9f48161e61c280f, a92dc402bb764b9bbb8bb5a2c7b38cdf; no voters:
I20260927 16:37:25.235954 17909 raft_consensus.cc:2781] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:25.236187 17909 raft_consensus.cc:686] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 1 LEADER]: Becoming Leader. State: Replica: a92dc402bb764b9bbb8bb5a2c7b38cdf, State: Running, Role: LEADER
I20260927 16:37:25.236514 17909 consensus_queue.cc:237] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } }
I20260927 16:37:25.239629 17424 catalog_manager.cc:5495] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to a92dc402bb764b9bbb8bb5a2c7b38cdf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a92dc402bb764b9bbb8bb5a2c7b38cdf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } health_report { overall_health: HEALTHY } } }
I20260927 16:37:25.264591 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.261201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.0.98.3:41679)
0927 16:37:25.261324 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.261327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.261354 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:37:25.261429 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.261431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.261484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.261570 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:25.261881 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.262465 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.263275 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.263492 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.263568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.263824 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:25.264248 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:25.264354 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.264406 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.264453 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:25.270884 17570 raft_consensus.cc:1261] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 1 FOLLOWER]: Refusing update from remote peer a92dc402bb764b9bbb8bb5a2c7b38cdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:25.270884 17704 raft_consensus.cc:1261] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 1 FOLLOWER]: Refusing update from remote peer a92dc402bb764b9bbb8bb5a2c7b38cdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:25.271641 17909 consensus_queue.cc:1035] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf4961fe2df4264aabda2b4f9debb31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:25.271862 17911 consensus_queue.cc:1035] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee0415b32ce4acba9f48161e61c280f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:25.281924 17915 mvcc.cc:205] Tried to move back new op lower bound from 7333998777421262848 to 7333998777291288576. Current Snapshot: MvccSnapshot[applied={T|T < 7333998777421262848}]
I20260927 16:37:25.288062 17424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:25.288256 17424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:25.290619 17424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3134031da2d4a1cb29b81623a2ba0ec: Sending DeleteTablet for 3 replicas of tablet 45d1c03b8b9c4d4b9e825a9606287f20
I20260927 16:37:25.291481 17550 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53872
I20260927 16:37:25.291492 17684 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260927 16:37:25.291617 17818 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d1c03b8b9c4d4b9e825a9606287f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42132
I20260927 16:37:25.292413 17925 tablet_replica.cc:330] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f: stopping tablet replica
I20260927 16:37:25.292785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17487
I20260927 16:37:25.292791 17925 raft_consensus.cc:2229] T 45d1c03b8b9c4d4b9e825a9606287f20 P 3ee0415b32ce4acba9f48161e61c280f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:25.293747 17924 tablet_replica.cc:330] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: stopping tablet replica
I20260927 16:37:25.294358 17924 raft_consensus.cc:2229] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:37:25.294723 17924 raft_consensus.cc:2258] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:25.295104 17926 tablet_replica.cc:330] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31: stopping tablet replica
I20260927 16:37:25.295373 17926 raft_consensus.cc:2229] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:25.295709 17926 raft_consensus.cc:2258] T 45d1c03b8b9c4d4b9e825a9606287f20 P caf4961fe2df4264aabda2b4f9debb31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:25.297082 17924 ts_tablet_manager.cc:1905] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:37:25.299600 17410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42775 (error 108)
I20260927 16:37:25.299948 17924 ts_tablet_manager.cc:1918] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:25.300046 17924 log.cc:1196] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/45d1c03b8b9c4d4b9e825a9606287f20
I20260927 16:37:25.300367 17924 ts_tablet_manager.cc:1939] T 45d1c03b8b9c4d4b9e825a9606287f20 P a92dc402bb764b9bbb8bb5a2c7b38cdf: Deleting consensus metadata
I20260927 16:37:25.300717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17621
I20260927 16:37:25.300776 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.300494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42775 (local address 127.0.0.1:53882)
0927 16:37:25.300604 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.300676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 16:37:25.302850 17412 catalog_manager.cc:4841] TS a92dc402bb764b9bbb8bb5a2c7b38cdf (127.0.98.3:41679): tablet 45d1c03b8b9c4d4b9e825a9606287f20 (table test-table [id=efc39faf4dd3492ca3041906627ad36d]) successfully deleted
W20260927 16:37:25.306370 17410 catalog_manager.cc:4584] TS 3ee0415b32ce4acba9f48161e61c280f (127.0.98.1:42775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45d1c03b8b9c4d4b9e825a9606287f20: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42775: connect: Connection refused (error 111)
I20260927 16:37:25.307152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17755
I20260927 16:37:25.309273 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.309007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37813 (local address 127.0.0.1:41456)
0927 16:37:25.309141 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.309201 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 16:37:25.312384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17393
2026-09-27T16:37:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2266 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T16:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:25Z Disabled control of system clock
I20260927 16:37:25.343398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42913
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:25.452929 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:25.453227 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:25.453262 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:25.457158 17931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:25.457212 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:25.457228 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:25.457244 17931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:25.457259 17931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:25.461912 17931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:25.463014 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:25.464323 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:25.472802 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.472855 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.472976 17931 server_base.cc:1034] running on GCE node
W20260927 16:37:25.472805 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.473714 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:25.474923 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:25.476084 17931 hybrid_clock.cc:648] HybridClock initialized: now 1790527045476051 us; error 49 us; skew 500 ppm
I20260927 16:37:25.478031 17931 webserver.cc:460] Webserver started at http://127.0.98.62:34313/ using document root <none> and password file <none>
I20260927 16:37:25.478696 17931 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:25.478763 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:25.479004 17931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:25.481014 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "23cde417245b310f1acff2056e5f19c0"
server_key_iv: "43a480d52e0cb4b52b3bfd93ae26acc0"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.481565 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "23cde417245b310f1acff2056e5f19c0"
server_key_iv: "43a480d52e0cb4b52b3bfd93ae26acc0"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.485138 17931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:37:25.487592 17945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.488883 17931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:25.489028 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "23cde417245b310f1acff2056e5f19c0"
server_key_iv: "43a480d52e0cb4b52b3bfd93ae26acc0"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.489137 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:25.511122 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:25.511863 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:25.512068 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:25.519187 17931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42913
I20260927 16:37:25.519210 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42913 every 8 connection(s)
I20260927 16:37:25.520145 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:25.523278 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:25.528649 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: Bootstrap starting.
I20260927 16:37:25.529095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 17931
I20260927 16:37:25.529227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:25.529496 392 external_mini_cluster.cc:1428] Setting key 09e7ce3d0e711b2530e5d82f447533ea
I20260927 16:37:25.531291 17998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:25.532035 17998 log.cc:824] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:25.534137 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: No bootstrap required, opened a new log
I20260927 16:37:25.538492 17998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260927 16:37:25.538741 17998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:25.538818 17998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc7e8c842ac46ffbdb7a336fba6a1c7, State: Initialized, Role: FOLLOWER
I20260927 16:37:25.539175 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.530805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.98.62:42913)
0927 16:37:25.531486 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.531502 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.531537 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:25.532224 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.532244 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.532747 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.533079 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:25.533430 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.534477 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.535747 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.536195 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.536312 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.537162 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:25.537167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:25.537179 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:25.537576 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:25.537628 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:25.537647 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:25.537722 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.537877 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.538206 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260927 16:37:25.539237 17998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260927 16:37:25.539415 17998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:25.539467 17998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:25.539533 17998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:25.540488 17998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260927 16:37:25.540843 17998 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fc7e8c842ac46ffbdb7a336fba6a1c7; no voters:
I20260927 16:37:25.541168 17998 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:25.541268 18003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:25.541496 18003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc7e8c842ac46ffbdb7a336fba6a1c7, State: Running, Role: LEADER
I20260927 16:37:25.541805 18003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260927 16:37:25.542447 17998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:25.543553 18005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc7e8c842ac46ffbdb7a336fba6a1c7. Latest consensus state: current_term: 1 leader_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } } }
I20260927 16:37:25.543707 18005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:25.543553 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc7e8c842ac46ffbdb7a336fba6a1c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } } }
I20260927 16:37:25.543869 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:25.544163 18012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:25.547137 18012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:25.552719 18012 catalog_manager.cc:1347] Generated new cluster ID: 7ff5c2c3bde04666a7c19bb5dd446320
I20260927 16:37:25.552807 18012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:25.577093 18012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:25.578131 18012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:25.586404 18012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: Generated new TSK 0
I20260927 16:37:25.587113 18012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:25.596535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--builtin_ntp_servers=127.0.98.20:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:25.709749 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:25.710045 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:25.710088 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:25.714251 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:25.714316 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:25.714423 18022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:25.719344 18022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:25.720479 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:25.721899 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:25.731757 18022 server_base.cc:1034] running on GCE node
W20260927 16:37:25.731703 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.731703 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.731699 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.732383 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:25.733052 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:25.734248 18022 hybrid_clock.cc:648] HybridClock initialized: now 1790527045734231 us; error 29 us; skew 500 ppm
I20260927 16:37:25.736107 18022 webserver.cc:460] Webserver started at http://127.0.98.1:42879/ using document root <none> and password file <none>
I20260927 16:37:25.736683 18022 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:25.736747 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:25.736987 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:25.739100 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "176f36c83b20481f893b757ef46de4f8"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "92548bb3c8a7100488b1e20f4c057bf8"
server_key_iv: "61a88ee73ea5f8e84dcf27fcc342a57b"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.739661 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "176f36c83b20481f893b757ef46de4f8"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "92548bb3c8a7100488b1e20f4c057bf8"
server_key_iv: "61a88ee73ea5f8e84dcf27fcc342a57b"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.743463 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 16:37:25.745994 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.747304 18022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:25.747432 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "176f36c83b20481f893b757ef46de4f8"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "92548bb3c8a7100488b1e20f4c057bf8"
server_key_iv: "61a88ee73ea5f8e84dcf27fcc342a57b"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.747545 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:25.774374 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:25.775141 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:25.775348 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:25.775974 18022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:25.776927 18022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:25.776974 18022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.777040 18022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:25.777071 18022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.785974 18022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43531
I20260927 16:37:25.786003 18149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43531 every 8 connection(s)
I20260927 16:37:25.787070 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:37:25.793198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18022
I20260927 16:37:25.793432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:25.793725 392 external_mini_cluster.cc:1428] Setting key b87ea199e28d3a2ea29bc825662f51d2
I20260927 16:37:25.795715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--builtin_ntp_servers=127.0.98.20:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:25.797453 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.788995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57005 (local address 127.0.98.62:42913)
0927 16:37:25.789213 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.789219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.790742 (+ 1523us) server_negotiation.cc:407] Connection header received
0927 16:37:25.792148 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.792154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.792213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.792300 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:25.793304 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.794017 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.795478 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.795710 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.795825 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.796951 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:25.796955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:25.796958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:25.796977 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:25.796996 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:25.797003 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:25.797079 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:25.797250 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 16:37:25.797310 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:25.798429 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.789655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.1:57005)
0927 16:37:25.790585 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.790628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.791916 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.792511 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.792519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.792605 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:25.793141 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.793154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.794158 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.794161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.795343 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.795352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.795863 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.795867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.796038 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:25.796685 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:25.796687 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:25.796729 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:25.796797 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:25.797093 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:25.797104 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:25.797112 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:37:25.797239 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 16:37:25.797549 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":191,"threads_started":1}
I20260927 16:37:25.799701 18150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260927 16:37:25.799978 18150 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:25.800675 18150 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260927 16:37:25.802323 17962 ts_manager.cc:194] Registered new tserver with Master: 176f36c83b20481f893b757ef46de4f8 (127.0.98.1:43531)
I20260927 16:37:25.803444 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57005
W20260927 16:37:25.908113 18153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:25.908452 18153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:25.908520 18153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:25.912830 18153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:25.912936 18153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:25.913041 18153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:25.918291 18153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:25.919483 18153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:25.920899 18153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:25.930429 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.930442 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:25.930439 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:25.930691 18153 server_base.cc:1034] running on GCE node
I20260927 16:37:25.931193 18153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:25.931839 18153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:25.933017 18153 hybrid_clock.cc:648] HybridClock initialized: now 1790527045932984 us; error 48 us; skew 500 ppm
I20260927 16:37:25.934978 18153 webserver.cc:460] Webserver started at http://127.0.98.2:43475/ using document root <none> and password file <none>
I20260927 16:37:25.935627 18153 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:25.935714 18153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:25.935949 18153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:25.938561 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba889f5c9176492eb1768c827c8f7072"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "6c437228dd85d82269a0e3cfc08259b1"
server_key_iv: "a249ff428cab62fca81037cce0af8c2c"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.939067 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba889f5c9176492eb1768c827c8f7072"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "6c437228dd85d82269a0e3cfc08259b1"
server_key_iv: "a249ff428cab62fca81037cce0af8c2c"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.942605 18153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:25.945070 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.946323 18153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:25.946457 18153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "ba889f5c9176492eb1768c827c8f7072"
format_stamp: "Formatted at 2026-09-27 16:37:25 on dist-test-slave-t27c"
server_key: "6c437228dd85d82269a0e3cfc08259b1"
server_key_iv: "a249ff428cab62fca81037cce0af8c2c"
server_key_version: "encryptionkey@0"
I20260927 16:37:25.946561 18153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:25.977794 18153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:25.978606 18153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:25.978825 18153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:25.979456 18153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:25.980414 18153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:25.980461 18153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.980501 18153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:25.980557 18153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:25.989389 18153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41605
I20260927 16:37:25.989410 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41605 every 8 connection(s)
I20260927 16:37:25.990393 18153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:25.993021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18153
I20260927 16:37:25.993153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:25.993501 392 external_mini_cluster.cc:1428] Setting key 46695802f7aff208438ac9e5eaa8739b
I20260927 16:37:25.995757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--builtin_ntp_servers=127.0.98.20:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:26.001160 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.992264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50329 (local address 127.0.98.62:42913)
0927 16:37:25.992432 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:25.992435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:25.993824 (+ 1389us) server_negotiation.cc:407] Connection header received
0927 16:37:25.995424 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:25.995426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:25.995470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:25.995539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:25.996788 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.997298 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.999117 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.999320 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.999400 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.000702 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:26.000705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:26.000707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:26.000719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:26.000737 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:26.000744 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:26.000806 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.001003 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.001051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:26.002516 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:25.992732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.2:50329)
0927 16:37:25.993643 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:25.993704 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:25.995189 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:25.995745 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:25.995755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:25.995887 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:25.996630 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.996652 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.997420 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.997426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.998999 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:25.999013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:25.999473 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:25.999480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:25.999714 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.000384 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:26.000388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:26.000450 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:26.000544 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:26.000831 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:26.000843 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:26.000852 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.001004 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.001334 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260927 16:37:26.003767 18281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260927 16:37:26.004009 18281 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:26.004519 18281 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260927 16:37:26.005739 17962 ts_manager.cc:194] Registered new tserver with Master: ba889f5c9176492eb1768c827c8f7072 (127.0.98.2:41605)
I20260927 16:37:26.006475 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50329
W20260927 16:37:26.110381 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:26.110724 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:26.110793 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:26.114970 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:26.115060 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:26.115152 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:26.119930 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:26.121021 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:26.122514 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:26.132086 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:26.132148 18284 server_base.cc:1034] running on GCE node
W20260927 16:37:26.132107 18289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:26.132097 18290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:26.132783 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:26.133395 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:26.134550 18284 hybrid_clock.cc:648] HybridClock initialized: now 1790527046134518 us; error 46 us; skew 500 ppm
I20260927 16:37:26.136288 18284 webserver.cc:460] Webserver started at http://127.0.98.3:39445/ using document root <none> and password file <none>
I20260927 16:37:26.136896 18284 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:26.136960 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:26.137197 18284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:26.139655 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "924a133b6a104d1fa37d7e055b8facf9"
format_stamp: "Formatted at 2026-09-27 16:37:26 on dist-test-slave-t27c"
server_key: "75768c44a7d6188da5ff2a9539871e62"
server_key_iv: "f66401836e5acaebd07172ca3af4d23a"
server_key_version: "encryptionkey@0"
I20260927 16:37:26.140213 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "924a133b6a104d1fa37d7e055b8facf9"
format_stamp: "Formatted at 2026-09-27 16:37:26 on dist-test-slave-t27c"
server_key: "75768c44a7d6188da5ff2a9539871e62"
server_key_iv: "f66401836e5acaebd07172ca3af4d23a"
server_key_version: "encryptionkey@0"
I20260927 16:37:26.144093 18284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:37:26.146597 18298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:26.147825 18284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:26.147944 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "924a133b6a104d1fa37d7e055b8facf9"
format_stamp: "Formatted at 2026-09-27 16:37:26 on dist-test-slave-t27c"
server_key: "75768c44a7d6188da5ff2a9539871e62"
server_key_iv: "f66401836e5acaebd07172ca3af4d23a"
server_key_version: "encryptionkey@0"
I20260927 16:37:26.148039 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:26.182408 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:26.183291 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:26.183454 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:26.184175 18284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:26.185166 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:26.185214 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:26.185252 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:26.185274 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:26.194628 18284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40833
I20260927 16:37:26.194703 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40833 every 8 connection(s)
I20260927 16:37:26.195611 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:26.202963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18284
I20260927 16:37:26.203099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:26.203370 392 external_mini_cluster.cc:1428] Setting key 5f5ca66e8dfc32a78fd500bf13ad3448
I20260927 16:37:26.204667 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.197460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57755 (local address 127.0.98.62:42913)
0927 16:37:26.197597 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.197600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.198744 (+ 1144us) server_negotiation.cc:407] Connection header received
0927 16:37:26.200085 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.200087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.200132 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.200208 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:26.201065 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.201568 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.202708 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.202930 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.203008 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.204123 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:26.204126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:26.204129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:26.204143 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:26.204167 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:26.204181 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:26.204273 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.204471 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.204530 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:26.205598 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.197906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.3:57755)
0927 16:37:26.198585 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.198627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.199900 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.200364 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.200370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.200455 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:26.200948 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.200960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.201663 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.201666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.202609 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.202618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.203088 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.203094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.203293 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.203916 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:26.203917 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:26.203957 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:26.204016 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:26.204281 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:26.204292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:26.204298 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.204470 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.204747 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
I20260927 16:37:26.206569 18412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260927 16:37:26.206800 18412 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:26.207293 18412 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260927 16:37:26.208230 17962 ts_manager.cc:194] Registered new tserver with Master: 924a133b6a104d1fa37d7e055b8facf9 (127.0.98.3:40833)
I20260927 16:37:26.208774 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57755
I20260927 16:37:26.217123 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:26.222468 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.219041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.0.98.62:42913)
0927 16:37:26.219175 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.219178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.219341 (+ 163us) server_negotiation.cc:407] Connection header received
0927 16:37:26.219447 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.219449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.219486 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.219551 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:26.219753 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.220267 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.221043 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.221241 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.221315 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.222001 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:26.222004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:26.222006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:26.222016 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:26.222033 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:26.222040 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:26.222097 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.222236 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.222283 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:37:26.224980 17962 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:26.228128 17962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:26.242995 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.237533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.98.3:40833)
0927 16:37:26.237904 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.237915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.237958 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:37:26.238119 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.238126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.238372 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.238548 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:26.239009 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.240225 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.241422 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242280 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.242393 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.242639 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.242711 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.242810 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":143,"threads_started":1}
I20260927 16:37:26.243026 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.236888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40833 (local address 127.0.0.1:35064)
0927 16:37:26.237846 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.237862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.237952 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.238714 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.238731 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.238761 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:26.238874 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.238883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.240385 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.240388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.241289 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.241302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242402 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.242414 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.242536 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.242575 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.242685 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.242834 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":4,"spinlock_wait_cycles":1664,"thread_start_us":102,"threads_started":1}
I20260927 16:37:26.243033 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.236824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43531 (local address 127.0.0.1:42086)
0927 16:37:26.237416 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.237468 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.237591 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.238974 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.238978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.238995 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:26.239070 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.239074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.240338 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.240341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.241503 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.241510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242547 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.242550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.242633 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.242655 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.242704 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.242840 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":76,"thread_start_us":108,"threads_started":1}
I20260927 16:37:26.243398 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.237559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41605 (local address 127.0.0.1:35256)
0927 16:37:26.238090 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.238103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.238203 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.238672 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.238678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.238699 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:26.238859 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.238870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.239993 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.239998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.241394 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.241404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242758 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.242763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.242843 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.242861 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.242925 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.243011 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":102,"spinlock_wait_cycles":4480,"thread_start_us":85,"threads_started":1}
I20260927 16:37:26.243909 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.237788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.0.98.2:41605)
0927 16:37:26.238133 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.238149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.238209 (+ 60us) server_negotiation.cc:407] Connection header received
0927 16:37:26.238314 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.238319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.238525 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.238680 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:26.239165 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.239875 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.241536 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242394 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.242509 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.242699 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.243664 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.243764 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260927 16:37:26.244117 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.236955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.98.1:43531)
0927 16:37:26.238537 (+ 1582us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.238548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.238569 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:37:26.238639 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.238644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.238834 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.239000 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:26.239132 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.240225 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.241620 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.242421 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.243681 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.243810 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.243850 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.243920 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":130,"threads_started":1}
I20260927 16:37:26.246445 18215 tablet_service.cc:1467] Processing CreateTablet for tablet f482b0face1b46e5becb3d53c2c32d26 (DEFAULT_TABLE table=test-table [id=11b825efd21146a6bd2194d91a06ac73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:26.246423 18346 tablet_service.cc:1467] Processing CreateTablet for tablet f482b0face1b46e5becb3d53c2c32d26 (DEFAULT_TABLE table=test-table [id=11b825efd21146a6bd2194d91a06ac73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:26.247574 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f482b0face1b46e5becb3d53c2c32d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:26.247574 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f482b0face1b46e5becb3d53c2c32d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:26.248008 18084 tablet_service.cc:1467] Processing CreateTablet for tablet f482b0face1b46e5becb3d53c2c32d26 (DEFAULT_TABLE table=test-table [id=11b825efd21146a6bd2194d91a06ac73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:26.249042 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f482b0face1b46e5becb3d53c2c32d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:26.255046 18430 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Bootstrap starting.
I20260927 16:37:26.255492 18429 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Bootstrap starting.
I20260927 16:37:26.256220 18431 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Bootstrap starting.
I20260927 16:37:26.257864 18431 tablet_bootstrap.cc:654] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:26.257869 18429 tablet_bootstrap.cc:654] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:26.258106 18430 tablet_bootstrap.cc:654] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:26.258666 18431 log.cc:824] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:26.258833 18430 log.cc:824] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:26.258757 18429 log.cc:824] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:26.260962 18431 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: No bootstrap required, opened a new log
I20260927 16:37:26.260962 18430 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: No bootstrap required, opened a new log
I20260927 16:37:26.261155 18430 ts_tablet_manager.cc:1397] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 16:37:26.261155 18431 ts_tablet_manager.cc:1397] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:37:26.261477 18429 tablet_bootstrap.cc:492] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: No bootstrap required, opened a new log
I20260927 16:37:26.261658 18429 ts_tablet_manager.cc:1397] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 16:37:26.264742 18429 raft_consensus.cc:348] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.264979 18429 raft_consensus.cc:374] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:26.265038 18429 raft_consensus.cc:729] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba889f5c9176492eb1768c827c8f7072, State: Initialized, Role: FOLLOWER
I20260927 16:37:26.265491 18429 consensus_queue.cc:260] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.265478 18430 raft_consensus.cc:348] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.265478 18431 raft_consensus.cc:348] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.265722 18431 raft_consensus.cc:374] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:26.265733 18430 raft_consensus.cc:374] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:26.265789 18431 raft_consensus.cc:729] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176f36c83b20481f893b757ef46de4f8, State: Initialized, Role: FOLLOWER
I20260927 16:37:26.265796 18430 raft_consensus.cc:729] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924a133b6a104d1fa37d7e055b8facf9, State: Initialized, Role: FOLLOWER
I20260927 16:37:26.266271 18430 consensus_queue.cc:260] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.266271 18431 consensus_queue.cc:260] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.266858 18429 ts_tablet_manager.cc:1428] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260927 16:37:26.266981 18281 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
I20260927 16:37:26.267231 18430 ts_tablet_manager.cc:1428] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:37:26.267238 18431 ts_tablet_manager.cc:1428] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:37:26.267555 18412 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
I20260927 16:37:26.268044 18150 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
W20260927 16:37:26.288244 18151 tablet.cc:2369] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:37:26.300428 18436 raft_consensus.cc:482] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:26.300626 18436 raft_consensus.cc:504] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.301733 18436 leader_election.cc:288] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176f36c83b20481f893b757ef46de4f8 (127.0.98.1:43531), ba889f5c9176492eb1768c827c8f7072 (127.0.98.2:41605)
I20260927 16:37:26.306692 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.302399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39709 (local address 127.0.98.2:41605)
0927 16:37:26.302523 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.302528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.302770 (+ 242us) server_negotiation.cc:407] Connection header received
0927 16:37:26.302857 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.302860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.302909 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.302991 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:26.303220 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.303924 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.305397 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.305996 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.306104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.306162 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.306497 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.306563 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:26.306692 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.302081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45319 (local address 127.0.98.1:43531)
0927 16:37:26.302260 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.302266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.302283 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:37:26.302561 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.302567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.302619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.302692 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:26.303009 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.303793 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.305403 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.305996 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.306104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.306161 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.306497 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.306563 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:26.306737 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.302017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43531 (local address 127.0.98.3:45319)
0927 16:37:26.302130 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.302146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.302272 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.302732 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.302738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.302758 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:26.302893 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.302899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.303891 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.303894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.305273 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.305283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.306120 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.306123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.306364 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.306400 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.306472 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.306572 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":170}
I20260927 16:37:26.306970 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.302297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41605 (local address 127.0.98.3:39709)
0927 16:37:26.302648 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.302660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:26.302773 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:26.303005 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:26.303008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:26.303030 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:26.303117 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.303121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.304009 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.304011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.305274 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:26.305283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.306130 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.306133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:26.306322 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.306387 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 16:37:26.306470 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:37:26.306572 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":111,"spinlock_wait_cycles":2432,"thread_start_us":166,"threads_started":1}
I20260927 16:37:26.307440 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482b0face1b46e5becb3d53c2c32d26" candidate_uuid: "924a133b6a104d1fa37d7e055b8facf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176f36c83b20481f893b757ef46de4f8" is_pre_election: true
I20260927 16:37:26.307616 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482b0face1b46e5becb3d53c2c32d26" candidate_uuid: "924a133b6a104d1fa37d7e055b8facf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba889f5c9176492eb1768c827c8f7072" is_pre_election: true
I20260927 16:37:26.307786 18104 raft_consensus.cc:2454] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924a133b6a104d1fa37d7e055b8facf9 in term 0.
I20260927 16:37:26.307858 18235 raft_consensus.cc:2454] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924a133b6a104d1fa37d7e055b8facf9 in term 0.
I20260927 16:37:26.308274 18299 leader_election.cc:302] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 924a133b6a104d1fa37d7e055b8facf9, ba889f5c9176492eb1768c827c8f7072; no voters:
I20260927 16:37:26.308519 18436 raft_consensus.cc:2781] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:26.308619 18436 raft_consensus.cc:482] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:26.308676 18436 raft_consensus.cc:3037] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:26.309691 18436 raft_consensus.cc:504] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.310043 18436 leader_election.cc:288] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [CANDIDATE]: Term 1 election: Requested vote from peers 176f36c83b20481f893b757ef46de4f8 (127.0.98.1:43531), ba889f5c9176492eb1768c827c8f7072 (127.0.98.2:41605)
I20260927 16:37:26.310431 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482b0face1b46e5becb3d53c2c32d26" candidate_uuid: "924a133b6a104d1fa37d7e055b8facf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176f36c83b20481f893b757ef46de4f8"
I20260927 16:37:26.310503 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482b0face1b46e5becb3d53c2c32d26" candidate_uuid: "924a133b6a104d1fa37d7e055b8facf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba889f5c9176492eb1768c827c8f7072"
I20260927 16:37:26.310575 18104 raft_consensus.cc:3037] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:26.310672 18235 raft_consensus.cc:3037] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:26.311640 18104 raft_consensus.cc:2454] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924a133b6a104d1fa37d7e055b8facf9 in term 1.
I20260927 16:37:26.311764 18235 raft_consensus.cc:2454] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924a133b6a104d1fa37d7e055b8facf9 in term 1.
I20260927 16:37:26.311976 18299 leader_election.cc:302] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176f36c83b20481f893b757ef46de4f8, 924a133b6a104d1fa37d7e055b8facf9; no voters:
I20260927 16:37:26.312178 18436 raft_consensus.cc:2781] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:26.312414 18436 raft_consensus.cc:686] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 1 LEADER]: Becoming Leader. State: Replica: 924a133b6a104d1fa37d7e055b8facf9, State: Running, Role: LEADER
I20260927 16:37:26.312791 18436 consensus_queue.cc:237] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } }
I20260927 16:37:26.315950 17962 catalog_manager.cc:5495] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 924a133b6a104d1fa37d7e055b8facf9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "924a133b6a104d1fa37d7e055b8facf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "924a133b6a104d1fa37d7e055b8facf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 } health_report { overall_health: UNKNOWN } } }
I20260927 16:37:26.340867 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.337360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.0.98.3:40833)
0927 16:37:26.337490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:26.337495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:26.337510 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:37:26.337623 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:26.337626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:26.337680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:26.337788 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:26.338023 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.338583 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.339435 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:26.339657 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:26.339761 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:26.340067 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:26.340485 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:26.340618 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:26.340654 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 16:37:26.340704 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:26.346973 18235 raft_consensus.cc:1261] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 1 FOLLOWER]: Refusing update from remote peer 924a133b6a104d1fa37d7e055b8facf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:26.347028 18104 raft_consensus.cc:1261] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 1 FOLLOWER]: Refusing update from remote peer 924a133b6a104d1fa37d7e055b8facf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:26.347728 18436 consensus_queue.cc:1035] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176f36c83b20481f893b757ef46de4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:26.347911 18439 consensus_queue.cc:1035] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba889f5c9176492eb1768c827c8f7072" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:26.354952 18442 mvcc.cc:205] Tried to move back new op lower bound from 7333998781830033408 to 7333998781700272128. Current Snapshot: MvccSnapshot[applied={T|T < 7333998781830033408}]
I20260927 16:37:26.356082 18443 mvcc.cc:205] Tried to move back new op lower bound from 7333998781830033408 to 7333998781700272128. Current Snapshot: MvccSnapshot[applied={T|T < 7333998781830033408}]
I20260927 16:37:26.363813 17962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:26.363996 17962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:26.366252 17962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fc7e8c842ac46ffbdb7a336fba6a1c7: Sending DeleteTablet for 3 replicas of tablet f482b0face1b46e5becb3d53c2c32d26
I20260927 16:37:26.367038 18346 tablet_service.cc:1514] Processing DeleteTablet for tablet f482b0face1b46e5becb3d53c2c32d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:26 UTC) from {username='slave'} at 127.0.0.1:35064
I20260927 16:37:26.367085 18084 tablet_service.cc:1514] Processing DeleteTablet for tablet f482b0face1b46e5becb3d53c2c32d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:26 UTC) from {username='slave'} at 127.0.0.1:42086
I20260927 16:37:26.367537 18452 tablet_replica.cc:330] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8: stopping tablet replica
I20260927 16:37:26.367542 18451 tablet_replica.cc:330] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: stopping tablet replica
I20260927 16:37:26.367575 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet f482b0face1b46e5becb3d53c2c32d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:26 UTC) from {username='slave'} at 127.0.0.1:35256
I20260927 16:37:26.367839 18451 raft_consensus.cc:2229] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:37:26.367854 18452 raft_consensus.cc:2229] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:26.368170 18451 raft_consensus.cc:2258] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:26.368184 18452 raft_consensus.cc:2258] T f482b0face1b46e5becb3d53c2c32d26 P 176f36c83b20481f893b757ef46de4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:26.368420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18022
I20260927 16:37:26.369865 18453 tablet_replica.cc:330] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: stopping tablet replica
I20260927 16:37:26.370216 18453 raft_consensus.cc:2229] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:26.370450 18451 ts_tablet_manager.cc:1905] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:26.370612 18453 raft_consensus.cc:2258] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:26.373168 18451 ts_tablet_manager.cc:1918] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:26.373281 18451 log.cc:1196] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/f482b0face1b46e5becb3d53c2c32d26
I20260927 16:37:26.373615 18451 ts_tablet_manager.cc:1939] T f482b0face1b46e5becb3d53c2c32d26 P 924a133b6a104d1fa37d7e055b8facf9: Deleting consensus metadata
I20260927 16:37:26.374759 17947 catalog_manager.cc:4841] TS 924a133b6a104d1fa37d7e055b8facf9 (127.0.98.3:40833): tablet f482b0face1b46e5becb3d53c2c32d26 (table test-table [id=11b825efd21146a6bd2194d91a06ac73]) successfully deleted
I20260927 16:37:26.375952 18453 ts_tablet_manager.cc:1905] T f482b0face1b46e5becb3d53c2c32d26 P ba889f5c9176492eb1768c827c8f7072: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:37:26.376865 17946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43531 (error 108)
I20260927 16:37:26.377664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18153
I20260927 16:37:26.377967 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.377631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43531 (local address 127.0.0.1:42094)
0927 16:37:26.377813 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.377891 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 16:37:26.378434 17946 catalog_manager.cc:4584] TS 176f36c83b20481f893b757ef46de4f8 (127.0.98.1:43531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f482b0face1b46e5becb3d53c2c32d26: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43531: connect: Connection refused (error 111)
I20260927 16:37:26.384827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18284
I20260927 16:37:26.386778 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:26.386421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41605 (local address 127.0.0.1:35266)
0927 16:37:26.386593 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:26.386664 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:37:26.390858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 17931
2026-09-27T16:37:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:26 dist-test-slave-t27c krb5kdc[18457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:26 dist-test-slave-t27c krb5kdc[18457](info): set up 2 sockets
Sep 27 16:37:26 dist-test-slave-t27c krb5kdc[18457](info): commencing operation
krb5kdc: starting...
W20260927 16:37:27.438382 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:27 dist-test-slave-t27c krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:27.558823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:27.669456 18470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:27.669752 18470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:27.669795 18470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:27.673897 18470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:27.674026 18470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:27.674063 18470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:27.674101 18470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:27.674134 18470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:27.679620 18470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:27.680671 18470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:27.681645 18470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:27.690363 18475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:27.690369 18476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:27.690428 18470 server_base.cc:1034] running on GCE node
W20260927 16:37:27.690369 18478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:27.691197 18470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:27.692301 18470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:27.693461 18470 hybrid_clock.cc:648] HybridClock initialized: now 1790527047693432 us; error 44 us; skew 500 ppm
Sep 27 16:37:27 dist-test-slave-t27c krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:27.696163 18470 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:27.697063 18470 webserver.cc:460] Webserver started at http://127.0.98.62:36167/ using document root <none> and password file <none>
I20260927 16:37:27.697660 18470 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:27.697764 18470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:27.697934 18470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:27.699929 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "7de9d13392fc4b729664890da306afc1"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "2cca650eadca0976d725ef6ce1654173"
server_key_iv: "728e6ccf64c7ea0526520856b2ce13c6"
server_key_version: "encryptionkey@0"
I20260927 16:37:27.700435 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "7de9d13392fc4b729664890da306afc1"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "2cca650eadca0976d725ef6ce1654173"
server_key_iv: "728e6ccf64c7ea0526520856b2ce13c6"
server_key_version: "encryptionkey@0"
I20260927 16:37:27.703975 18470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:37:27.706349 18485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:27.707574 18470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:27.707700 18470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "7de9d13392fc4b729664890da306afc1"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "2cca650eadca0976d725ef6ce1654173"
server_key_iv: "728e6ccf64c7ea0526520856b2ce13c6"
server_key_version: "encryptionkey@0"
I20260927 16:37:27.707810 18470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:27.723430 18470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:27.726310 18470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:27.726512 18470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:27.733482 18470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39531
I20260927 16:37:27.733501 18537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39531 every 8 connection(s)
I20260927 16:37:27.734478 18470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:27.735199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18470
I20260927 16:37:27.735334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:27.735664 392 external_mini_cluster.cc:1428] Setting key 06e04f2487e0235cfd0fc546cb4f6b59
I20260927 16:37:27.737732 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:27.743248 18538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: Bootstrap starting.
Sep 27 16:37:27 dist-test-slave-t27c krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:27.745826 18538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:27.746624 18538 log.cc:824] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:27.748931 18538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: No bootstrap required, opened a new log
I20260927 16:37:27.749940 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:27.737069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.98.62:39531)
0927 16:37:27.737487 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:27.737501 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:27.737536 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:27.738278 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:27.738295 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:27.738616 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:27.738965 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:27.739761 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:27.740751 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:27.741605 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:27.742066 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:27.742260 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:27.744937 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:27.744943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:27.744956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:27.744983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:27.746805 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:27.747307 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:27.747312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:27.747316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:27.747477 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:27.747749 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:27.747752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:27.747754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:27.748245 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:27.748438 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:27.748677 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 16:37:27.748985 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260927 16:37:27.753115 18538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 16:37:27.753347 18538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:27.753381 18538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de9d13392fc4b729664890da306afc1, State: Initialized, Role: FOLLOWER
I20260927 16:37:27.753746 18538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 16:37:27.753850 18538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:27.753895 18538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:27.753965 18538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:27.754966 18538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 16:37:27.755283 18538 leader_election.cc:302] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de9d13392fc4b729664890da306afc1; no voters:
I20260927 16:37:27.755548 18538 leader_election.cc:288] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:27.755697 18543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:27.755949 18543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [term 1 LEADER]: Becoming Leader. State: Replica: 7de9d13392fc4b729664890da306afc1, State: Running, Role: LEADER
I20260927 16:37:27.756237 18543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 16:37:27.756788 18538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:27.757929 18545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de9d13392fc4b729664890da306afc1. Latest consensus state: current_term: 1 leader_uuid: "7de9d13392fc4b729664890da306afc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } } }
I20260927 16:37:27.758055 18545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:27.758352 18544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de9d13392fc4b729664890da306afc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de9d13392fc4b729664890da306afc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } } }
I20260927 16:37:27.758458 18544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:27.758533 18552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:27.761701 18552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:27.769871 18552 catalog_manager.cc:1347] Generated new cluster ID: 80fb46ccebed4551b6c389fd8be2b262
I20260927 16:37:27.769945 18552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:27.790933 18552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:27.792009 18552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:27.800894 18552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: Generated new TSK 0
I20260927 16:37:27.801651 18552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:37:27.841436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:27.965405 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:27.965715 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:27.965794 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:27.969926 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:27.970021 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:27.970114 18564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:27.975476 18564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:27.976567 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:27.977672 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:27.987329 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:27.987323 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:27.987323 18564 server_base.cc:1034] running on GCE node
W20260927 16:37:27.987341 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:27.987999 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:27.988633 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:27.989820 18564 hybrid_clock.cc:648] HybridClock initialized: now 1790527047989803 us; error 29 us; skew 500 ppm
Sep 27 16:37:27 dist-test-slave-t27c krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:27.992595 18564 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:37:27.993477 18564 webserver.cc:460] Webserver started at http://127.0.98.1:37121/ using document root <none> and password file <none>
I20260927 16:37:27.994063 18564 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:27.994153 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:27.994395 18564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:27.996435 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "43844bbeffa13ceb0bf99482d0a923e5"
server_key_iv: "400e8ba6ce7711ee061ad233d27fae68"
server_key_version: "encryptionkey@0"
I20260927 16:37:27.997004 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "43844bbeffa13ceb0bf99482d0a923e5"
server_key_iv: "400e8ba6ce7711ee061ad233d27fae68"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.000674 18564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:37:28.003046 18579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.004238 18564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:28.004370 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac"
format_stamp: "Formatted at 2026-09-27 16:37:27 on dist-test-slave-t27c"
server_key: "43844bbeffa13ceb0bf99482d0a923e5"
server_key_iv: "400e8ba6ce7711ee061ad233d27fae68"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.004498 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:28.028615 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:28.031792 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:28.031973 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:28.032517 18564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:28.033311 18564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:28.033358 18564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.033397 18564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:28.033421 18564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.042052 18564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42587
I20260927 16:37:28.042086 18692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42587 every 8 connection(s)
I20260927 16:37:28.042969 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:37:28.048764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18564
I20260927 16:37:28.048902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:28.049232 392 external_mini_cluster.cc:1428] Setting key 69ae6194d58b16c121d3bea8fa8309cf
Sep 27 16:37:28 dist-test-slave-t27c krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:28.057366 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.044744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52345 (local address 127.0.98.62:39531)
0927 16:37:28.044890 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.044895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.045793 (+ 898us) server_negotiation.cc:407] Connection header received
0927 16:37:28.046690 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.046693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.046739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.046813 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:28.047914 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.048439 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.049966 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.050320 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.050434 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.053181 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:28.053188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:28.053193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:28.053228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:28.055072 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.055605 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.055610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.055613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.055710 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.056045 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.056050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.056054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.056359 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:28.056479 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.056943 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.057106 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":55}
I20260927 16:37:28.058288 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.045170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.1:52345)
0927 16:37:28.045632 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.045675 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.046503 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.046974 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.046981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.047319 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:28.047792 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.047804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.048564 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.048569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.049831 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.049840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.050474 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.050478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.050685 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.051291 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:28.051316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:28.053017 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:28.055212 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.055220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.055242 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.055502 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.055810 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.055812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.055814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.055943 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.056472 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:28.056476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:28.056702 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.056858 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.057111 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:37:28.059610 18693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 16:37:28.059859 18693 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:28.060379 18693 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 16:37:28.061905 18502 ts_manager.cc:194] Registered new tserver with Master: 8b7e95ee02d74801a8ebbdfa32c9f8ac (127.0.98.1:42587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:37:28.063126 18502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:37:28.077833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:28.194023 18698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:28.194375 18698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:28.194458 18698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:28.198901 18698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:28.198962 18698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:28.199064 18698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:28.204507 18698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:28.205643 18698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:28.206785 18698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:28.216533 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:28.216626 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:28.216625 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:28.216753 18698 server_base.cc:1034] running on GCE node
I20260927 16:37:28.217166 18698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:28.217778 18698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:28.218936 18698 hybrid_clock.cc:648] HybridClock initialized: now 1790527048218910 us; error 45 us; skew 500 ppm
Sep 27 16:37:28 dist-test-slave-t27c krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:28.227425 18698 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:37:28.228413 18698 webserver.cc:460] Webserver started at http://127.0.98.2:36731/ using document root <none> and password file <none>
I20260927 16:37:28.229013 18698 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:28.229102 18698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:28.229339 18698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:28.231628 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2218ec979484d858ce0d2ba5f14beac"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "ba1703e4cdd9a07ab0496f7e10e0bf5c"
server_key_iv: "de1c28457455d656b586fcd41520f4e8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.232206 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2218ec979484d858ce0d2ba5f14beac"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "ba1703e4cdd9a07ab0496f7e10e0bf5c"
server_key_iv: "de1c28457455d656b586fcd41520f4e8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.236017 18698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:37:28.238574 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.239789 18698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:28.239921 18698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "d2218ec979484d858ce0d2ba5f14beac"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "ba1703e4cdd9a07ab0496f7e10e0bf5c"
server_key_iv: "de1c28457455d656b586fcd41520f4e8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.240032 18698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:28.254984 18698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:28.257880 18698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:28.258068 18698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:28.258661 18698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:28.259519 18698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:28.259565 18698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.259634 18698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:28.259678 18698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.268513 18698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39195
I20260927 16:37:28.268530 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39195 every 8 connection(s)
I20260927 16:37:28.269423 18698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:28.277186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18698
I20260927 16:37:28.277347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:28.277647 392 external_mini_cluster.cc:1428] Setting key 903d29cee7f38a509a6345543aca9576
Sep 27 16:37:28 dist-test-slave-t27c krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:28.283023 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.271331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34863 (local address 127.0.98.62:39531)
0927 16:37:28.271517 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.271522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.272372 (+ 850us) server_negotiation.cc:407] Connection header received
0927 16:37:28.273210 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.273213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.273258 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.273333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:28.274554 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.275079 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.276179 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.276386 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.276469 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.279309 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:28.279312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:28.279314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:28.279335 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:28.280814 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.281312 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.281318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.281320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.281414 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.281734 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.281739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.281742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.281978 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:28.282061 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.282617 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.282763 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:37:28.283696 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.271768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.2:34863)
0927 16:37:28.272213 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.272255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.273024 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.273505 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.273512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.273868 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:28.274420 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.274435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.275182 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.275184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.276077 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.276085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.276526 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.276529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.276666 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.277346 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:28.277363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:28.279148 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:28.280938 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.280942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.280954 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.281213 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.281511 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.281513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.281515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.281641 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.282072 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:28.282076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:28.282370 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.282536 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.282789 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260927 16:37:28.285066 18827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 16:37:28.285308 18827 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:28.285799 18827 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 16:37:28.286969 18502 ts_manager.cc:194] Registered new tserver with Master: d2218ec979484d858ce0d2ba5f14beac (127.0.98.2:39195)
I20260927 16:37:28.287846 18502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34863
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:37:28.305902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:28.417505 18832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:28.417807 18832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:28.417886 18832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:28.421921 18832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:28.422011 18832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:28.422111 18832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:28.427398 18832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:28.428522 18832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:28.429620 18832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:28.439818 18838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:28.439697 18837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:28.439693 18840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:28.440246 18832 server_base.cc:1034] running on GCE node
I20260927 16:37:28.440654 18832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:28.441332 18832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:28.442518 18832 hybrid_clock.cc:648] HybridClock initialized: now 1790527048442503 us; error 27 us; skew 500 ppm
Sep 27 16:37:28 dist-test-slave-t27c krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:28.445282 18832 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:37:28.446329 18832 webserver.cc:460] Webserver started at http://127.0.98.3:42573/ using document root <none> and password file <none>
I20260927 16:37:28.446910 18832 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:28.447010 18832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:28.447255 18832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:28.449335 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "dc4c70791fa659e6822a4bb167b34925"
server_key_iv: "6d212cf06cbe4394436b7ea8ae3f363e"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.449906 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "dc4c70791fa659e6822a4bb167b34925"
server_key_iv: "6d212cf06cbe4394436b7ea8ae3f363e"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.453624 18832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:37:28.456099 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.457473 18832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:28.457602 18832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "dc4c70791fa659e6822a4bb167b34925"
server_key_iv: "6d212cf06cbe4394436b7ea8ae3f363e"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.457715 18832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:28.475107 18832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:28.478129 18832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:28.478389 18832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:28.479029 18832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:28.479887 18832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:28.479960 18832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.480041 18832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:28.480093 18832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.489218 18832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44563
I20260927 16:37:28.489238 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44563 every 8 connection(s)
I20260927 16:37:28.490134 18832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:28.492100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 18832
I20260927 16:37:28.492256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:28.492532 392 external_mini_cluster.cc:1428] Setting key f6665a53358c73cca800619b4d99630f
Sep 27 16:37:28 dist-test-slave-t27c krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:28.503679 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.491962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47863 (local address 127.0.98.62:39531)
0927 16:37:28.492125 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.492130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.493138 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 16:37:28.494400 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.494403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.494460 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.494546 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:28.495807 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.496348 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.497532 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.497767 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.497878 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.500452 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:28.500455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:28.500457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:28.500483 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:28.501718 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.502240 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.502243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.502245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.502356 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.502729 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.502732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.502733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.502915 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:28.502994 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.503450 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.503554 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 16:37:28.504537 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.492453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.3:47863)
0927 16:37:28.492982 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.493024 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.494135 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.494735 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.494762 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.495168 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:28.495660 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.495674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.496481 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.496485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.497397 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.497406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.497918 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.497922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.498105 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.498722 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:28.498738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:28.500283 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:28.501823 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.501827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.501841 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.502097 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.502461 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:28.502464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:28.502466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:28.502630 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:28.503018 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:28.503023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:28.503250 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.503405 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.503659 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":147,"threads_started":1}
I20260927 16:37:28.505858 18961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 16:37:28.506134 18961 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:28.506771 18961 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 16:37:28.507768 18502 ts_manager.cc:194] Registered new tserver with Master: ee5df2aa1ee04f54bc3a12a3cc259acb (127.0.98.3:44563)
I20260927 16:37:28.508407 18502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47863
I20260927 16:37:28.518551 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:28.527606 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.520500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.98.62:39531)
0927 16:37:28.520622 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.520626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.520802 (+ 176us) server_negotiation.cc:407] Connection header received
0927 16:37:28.520903 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.520905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.520945 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.521009 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:28.521634 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.522144 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.523009 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.523224 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.523301 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.524837 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:28.524840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:28.524842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:28.524861 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:28.526074 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.526562 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.526565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.526567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.526636 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:28.526892 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:28.526894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:28.526895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:28.527084 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:28.527196 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.527401 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.527476 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:37:28.530622 18502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:28.533653 18502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:28.548837 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.542321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39195 (local address 127.0.0.1:37678)
0927 16:37:28.543873 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.543888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.543970 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.544492 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.544495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.544514 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:28.544601 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.544606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545940 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.545944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.546809 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.546818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.547922 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.547926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.548205 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.548354 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.548481 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.548659 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"mutex_wait_us":182,"thread_start_us":88,"threads_started":1}
I20260927 16:37:28.548833 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.542747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.98.2:39195)
0927 16:37:28.543212 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.543229 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.544077 (+ 848us) server_negotiation.cc:407] Connection header received
0927 16:37:28.544153 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.544158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.544336 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.544484 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:28.544728 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545792 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.546946 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.547795 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.547913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.548064 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.548534 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.548670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":108,"threads_started":1}
I20260927 16:37:28.549346 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.542738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.0.98.1:42587)
0927 16:37:28.543140 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.543161 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.543186 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:37:28.543269 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.543275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.543457 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.543623 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:28.543937 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545196 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.547017 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.547795 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.547913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.548064 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.549074 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.549186 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":82,"threads_started":1}
I20260927 16:37:28.549579 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.542085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42587 (local address 127.0.0.1:59182)
0927 16:37:28.543065 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.543078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.543154 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.543622 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.543628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.543650 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:28.543794 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.543802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545324 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.545327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.546902 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.546911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.547922 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.547926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.548207 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.548354 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.548474 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.549455 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":1194,"thread_start_us":127,"threads_started":1}
I20260927 16:37:28.549777 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.541923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44563 (local address 127.0.0.1:39028)
0927 16:37:28.542614 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.542665 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.542787 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.543777 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.543780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.543797 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:28.543912 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.543922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545144 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.545149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.546900 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.546911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.549143 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.549147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.549246 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.549274 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.549324 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.549552 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":122,"thread_start_us":95,"threads_started":1}
I20260927 16:37:28.550258 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.542111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.98.3:44563)
0927 16:37:28.543411 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.543420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.543439 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:37:28.543501 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.543504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.543648 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.543797 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:28.544157 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.545021 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.548126 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.548824 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.549757 (+ 933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.549931 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.549986 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.550090 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":142,"threads_started":1}
I20260927 16:37:28.552645 18761 tablet_service.cc:1467] Processing CreateTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 (DEFAULT_TABLE table=test-table [id=4e6ed3cb219847d0a187adb21e5e5c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:28.552645 18627 tablet_service.cc:1467] Processing CreateTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 (DEFAULT_TABLE table=test-table [id=4e6ed3cb219847d0a187adb21e5e5c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:28.552721 18895 tablet_service.cc:1467] Processing CreateTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 (DEFAULT_TABLE table=test-table [id=4e6ed3cb219847d0a187adb21e5e5c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:28.553725 18627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acbc891cc5a45b991fafece75f9ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:28.553740 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acbc891cc5a45b991fafece75f9ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:28.553740 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acbc891cc5a45b991fafece75f9ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:28.560158 18978 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: Bootstrap starting.
I20260927 16:37:28.561570 18980 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Bootstrap starting.
I20260927 16:37:28.561980 18979 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Bootstrap starting.
I20260927 16:37:28.562723 18978 tablet_bootstrap.cc:654] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:28.563596 18980 tablet_bootstrap.cc:654] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:28.563650 18978 log.cc:824] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:28.564085 18979 tablet_bootstrap.cc:654] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:28.564723 18980 log.cc:824] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:28.565214 18979 log.cc:824] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:28.565828 18978 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: No bootstrap required, opened a new log
I20260927 16:37:28.566051 18978 ts_tablet_manager.cc:1397] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:37:28.566515 18980 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: No bootstrap required, opened a new log
I20260927 16:37:28.566708 18980 ts_tablet_manager.cc:1397] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:37:28.567096 18979 tablet_bootstrap.cc:492] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: No bootstrap required, opened a new log
I20260927 16:37:28.567284 18979 ts_tablet_manager.cc:1397] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:37:28.569583 18980 raft_consensus.cc:348] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.569828 18980 raft_consensus.cc:374] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:28.569890 18980 raft_consensus.cc:729] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2218ec979484d858ce0d2ba5f14beac, State: Initialized, Role: FOLLOWER
I20260927 16:37:28.570376 18980 consensus_queue.cc:260] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.570451 18978 raft_consensus.cc:348] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.570696 18978 raft_consensus.cc:374] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:28.570746 18978 raft_consensus.cc:729] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7e95ee02d74801a8ebbdfa32c9f8ac, State: Initialized, Role: FOLLOWER
I20260927 16:37:28.571020 18979 raft_consensus.cc:348] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.571182 18978 consensus_queue.cc:260] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.571275 18979 raft_consensus.cc:374] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:28.571323 18979 raft_consensus.cc:729] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5df2aa1ee04f54bc3a12a3cc259acb, State: Initialized, Role: FOLLOWER
I20260927 16:37:28.571379 18980 ts_tablet_manager.cc:1428] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:37:28.571722 18827 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 16:37:28.571719 18979 consensus_queue.cc:260] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.572165 18978 ts_tablet_manager.cc:1428] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:37:28.572335 18961 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 16:37:28.572467 18693 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 16:37:28.572629 18979 ts_tablet_manager.cc:1428] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:37:28.639611 18985 raft_consensus.cc:482] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:28.639843 18985 raft_consensus.cc:504] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.641023 18985 leader_election.cc:288] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5df2aa1ee04f54bc3a12a3cc259acb (127.0.98.3:44563), d2218ec979484d858ce0d2ba5f14beac (127.0.98.2:39195)
I20260927 16:37:28.645504 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.641446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51159 (local address 127.0.98.2:39195)
0927 16:37:28.641592 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.641597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.641969 (+ 372us) server_negotiation.cc:407] Connection header received
0927 16:37:28.642110 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.642113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.642162 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.642252 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:28.642537 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.643055 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.644405 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.644880 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.645023 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.645091 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.645285 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.645370 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:37:28.645527 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.641300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39195 (local address 127.0.98.1:51159)
0927 16:37:28.641859 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.641875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.641983 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.642281 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.642283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.642298 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:28.642422 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.642428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.643174 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.643177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.644203 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.644212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.645033 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.645037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.645138 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.645162 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.645269 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.645342 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":136,"thread_start_us":88,"threads_started":1}
I20260927 16:37:28.645529 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.641520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53665 (local address 127.0.98.3:44563)
0927 16:37:28.641645 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.641651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.641824 (+ 173us) server_negotiation.cc:407] Connection header received
0927 16:37:28.641934 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.641940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.641996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.642076 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:28.642420 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.642974 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.644450 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.644962 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.645083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.645149 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.645334 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.645408 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:28.645541 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.641300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44563 (local address 127.0.98.1:53665)
0927 16:37:28.641728 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.641740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:28.641840 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:28.642098 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:28.642101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:28.642118 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:28.642302 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.642309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.643096 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.643102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.644255 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:28.644305 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.645090 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.645094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:28.645199 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.645220 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:37:28.645317 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 16:37:28.645387 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":92,"thread_start_us":133,"threads_started":1}
I20260927 16:37:28.646338 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acbc891cc5a45b991fafece75f9ed93" candidate_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2218ec979484d858ce0d2ba5f14beac" is_pre_election: true
I20260927 16:37:28.646461 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acbc891cc5a45b991fafece75f9ed93" candidate_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" is_pre_election: true
I20260927 16:37:28.646708 18781 raft_consensus.cc:2454] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7e95ee02d74801a8ebbdfa32c9f8ac in term 0.
I20260927 16:37:28.646726 18915 raft_consensus.cc:2454] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7e95ee02d74801a8ebbdfa32c9f8ac in term 0.
I20260927 16:37:28.647294 18580 leader_election.cc:302] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b7e95ee02d74801a8ebbdfa32c9f8ac, d2218ec979484d858ce0d2ba5f14beac, ee5df2aa1ee04f54bc3a12a3cc259acb; no voters:
I20260927 16:37:28.647607 18985 raft_consensus.cc:2781] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:28.647710 18985 raft_consensus.cc:482] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:28.647773 18985 raft_consensus.cc:3037] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:28.648828 18985 raft_consensus.cc:504] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.649199 18985 leader_election.cc:288] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [CANDIDATE]: Term 1 election: Requested vote from peers ee5df2aa1ee04f54bc3a12a3cc259acb (127.0.98.3:44563), d2218ec979484d858ce0d2ba5f14beac (127.0.98.2:39195)
I20260927 16:37:28.649597 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acbc891cc5a45b991fafece75f9ed93" candidate_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2218ec979484d858ce0d2ba5f14beac"
I20260927 16:37:28.649598 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acbc891cc5a45b991fafece75f9ed93" candidate_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb"
I20260927 16:37:28.649724 18915 raft_consensus.cc:3037] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:28.649724 18781 raft_consensus.cc:3037] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:28.651043 18915 raft_consensus.cc:2454] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7e95ee02d74801a8ebbdfa32c9f8ac in term 1.
I20260927 16:37:28.651043 18781 raft_consensus.cc:2454] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7e95ee02d74801a8ebbdfa32c9f8ac in term 1.
I20260927 16:37:28.651364 18582 leader_election.cc:302] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b7e95ee02d74801a8ebbdfa32c9f8ac, d2218ec979484d858ce0d2ba5f14beac, ee5df2aa1ee04f54bc3a12a3cc259acb; no voters:
I20260927 16:37:28.651612 18985 raft_consensus.cc:2781] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:28.651818 18985 raft_consensus.cc:686] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [term 1 LEADER]: Becoming Leader. State: Replica: 8b7e95ee02d74801a8ebbdfa32c9f8ac, State: Running, Role: LEADER
I20260927 16:37:28.652145 18985 consensus_queue.cc:237] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } }
I20260927 16:37:28.655174 18502 catalog_manager.cc:5495] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7e95ee02d74801a8ebbdfa32c9f8ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7e95ee02d74801a8ebbdfa32c9f8ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 } health_report { overall_health: UNKNOWN } } }
I20260927 16:37:28.669840 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.666300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.98.1:42587)
0927 16:37:28.666433 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.666438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.666455 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:37:28.666556 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.666564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.666616 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.666731 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:28.666979 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.667497 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.668334 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.668572 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.668677 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.668991 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:28.669395 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:28.669567 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.669652 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.669689 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:37:28.675935 18781 raft_consensus.cc:1261] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 1 FOLLOWER]: Refusing update from remote peer 8b7e95ee02d74801a8ebbdfa32c9f8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:28.675935 18915 raft_consensus.cc:1261] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 1 FOLLOWER]: Refusing update from remote peer 8b7e95ee02d74801a8ebbdfa32c9f8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:28.676671 18985 consensus_queue.cc:1035] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5df2aa1ee04f54bc3a12a3cc259acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:28.676870 18989 consensus_queue.cc:1035] T 9acbc891cc5a45b991fafece75f9ed93 P 8b7e95ee02d74801a8ebbdfa32c9f8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2218ec979484d858ce0d2ba5f14beac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:28.685526 18992 mvcc.cc:205] Tried to move back new op lower bound from 7333998791368851456 to 7333998791281643520. Current Snapshot: MvccSnapshot[applied={T|T < 7333998791368851456}]
I20260927 16:37:28.693892 18500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:28.694108 18500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:28.696640 18500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de9d13392fc4b729664890da306afc1: Sending DeleteTablet for 3 replicas of tablet 9acbc891cc5a45b991fafece75f9ed93
I20260927 16:37:28.697435 18627 tablet_service.cc:1514] Processing DeleteTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59182
I20260927 16:37:28.697409 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260927 16:37:28.697602 18761 tablet_service.cc:1514] Processing DeleteTablet for tablet 9acbc891cc5a45b991fafece75f9ed93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37678
I20260927 16:37:28.697955 19003 tablet_replica.cc:330] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: stopping tablet replica
I20260927 16:37:28.698366 19003 raft_consensus.cc:2229] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:28.698395 19004 tablet_replica.cc:330] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: stopping tablet replica
I20260927 16:37:28.698670 19004 raft_consensus.cc:2229] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:28.698702 19003 raft_consensus.cc:2258] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:28.698959 19004 raft_consensus.cc:2258] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:28.699008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18564
I20260927 16:37:28.701210 19004 ts_tablet_manager.cc:1905] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:28.702061 19003 ts_tablet_manager.cc:1905] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:28.703606 19004 ts_tablet_manager.cc:1918] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:28.703714 19004 log.cc:1196] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/9acbc891cc5a45b991fafece75f9ed93
I20260927 16:37:28.704042 19004 ts_tablet_manager.cc:1939] T 9acbc891cc5a45b991fafece75f9ed93 P d2218ec979484d858ce0d2ba5f14beac: Deleting consensus metadata
I20260927 16:37:28.705158 18488 catalog_manager.cc:4841] TS d2218ec979484d858ce0d2ba5f14beac (127.0.98.2:39195): tablet 9acbc891cc5a45b991fafece75f9ed93 (table test-table [id=4e6ed3cb219847d0a187adb21e5e5c49]) successfully deleted
I20260927 16:37:28.705420 19003 ts_tablet_manager.cc:1918] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:28.705525 19003 log.cc:1196] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/9acbc891cc5a45b991fafece75f9ed93
I20260927 16:37:28.705853 19003 ts_tablet_manager.cc:1939] T 9acbc891cc5a45b991fafece75f9ed93 P ee5df2aa1ee04f54bc3a12a3cc259acb: Deleting consensus metadata
W20260927 16:37:28.707021 18487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42587 (error 108)
I20260927 16:37:28.707237 18486 catalog_manager.cc:4841] TS ee5df2aa1ee04f54bc3a12a3cc259acb (127.0.98.3:44563): tablet 9acbc891cc5a45b991fafece75f9ed93 (table test-table [id=4e6ed3cb219847d0a187adb21e5e5c49]) successfully deleted
I20260927 16:37:28.707876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18698
I20260927 16:37:28.708055 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.707726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42587 (local address 127.0.0.1:59200)
0927 16:37:28.707873 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:28.707952 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 16:37:28.708550 18487 catalog_manager.cc:4584] TS 8b7e95ee02d74801a8ebbdfa32c9f8ac (127.0.98.1:42587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9acbc891cc5a45b991fafece75f9ed93: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42587: connect: Connection refused (error 111)
I20260927 16:37:28.714012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18832
I20260927 16:37:28.719740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 18470
2026-09-27T16:37:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T16:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:28Z Disabled control of system clock
I20260927 16:37:28.749975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:28.862147 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:28.862439 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:28.862476 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:28.866462 19009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:28.866514 19009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:28.866529 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:28.866545 19009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:28.866560 19009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:28.871249 19009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:28.872356 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:28.873775 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:28.882468 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:28.882483 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:28.882586 19009 server_base.cc:1034] running on GCE node
W20260927 16:37:28.882474 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:28.883325 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:28.884362 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:28.885526 19009 hybrid_clock.cc:648] HybridClock initialized: now 1790527048885504 us; error 26 us; skew 500 ppm
I20260927 16:37:28.887354 19009 webserver.cc:460] Webserver started at http://127.0.98.62:42519/ using document root <none> and password file <none>
I20260927 16:37:28.887943 19009 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:28.888005 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:28.888231 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:28.890349 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "a8a9e72891ec410ba901705dfe969cc8"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "e742f2ccfa2f4e84697196442d774df1"
server_key_iv: "5fd343402d3ff6c979df44b383fb5db8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.890931 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8a9e72891ec410ba901705dfe969cc8"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "e742f2ccfa2f4e84697196442d774df1"
server_key_iv: "5fd343402d3ff6c979df44b383fb5db8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.894646 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:37:28.897032 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:28.898303 19009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:28.898432 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "a8a9e72891ec410ba901705dfe969cc8"
format_stamp: "Formatted at 2026-09-27 16:37:28 on dist-test-slave-t27c"
server_key: "e742f2ccfa2f4e84697196442d774df1"
server_key_iv: "5fd343402d3ff6c979df44b383fb5db8"
server_key_version: "encryptionkey@0"
I20260927 16:37:28.898545 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:28.916081 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:28.916911 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:28.917130 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:28.924353 19009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41043
I20260927 16:37:28.924382 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41043 every 8 connection(s)
I20260927 16:37:28.925309 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:28.925800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19009
I20260927 16:37:28.925946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:28.926297 392 external_mini_cluster.cc:1428] Setting key cd68d8e6d00564ae435bbc6e075d67db
I20260927 16:37:28.929044 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:28.935829 19076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: Bootstrap starting.
I20260927 16:37:28.936666 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:28.927775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.98.62:41043)
0927 16:37:28.928426 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:28.928446 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:28.928491 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:37:28.929361 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:28.929384 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:28.929882 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:28.930265 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:28.930714 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.932164 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.932995 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:28.933659 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:28.933835 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:28.934678 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:28.934684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:28.934699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:28.935190 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:28.935258 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:28.935284 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:28.935385 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:28.935483 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 16:37:28.935780 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260927 16:37:28.938153 19076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:28.938970 19076 log.cc:824] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:28.940840 19076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: No bootstrap required, opened a new log
I20260927 16:37:28.944248 19076 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } }
I20260927 16:37:28.944463 19076 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:28.944542 19076 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a9e72891ec410ba901705dfe969cc8, State: Initialized, Role: FOLLOWER
I20260927 16:37:28.944941 19076 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } }
I20260927 16:37:28.945088 19076 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:28.945151 19076 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:28.945266 19076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:28.946259 19076 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } }
I20260927 16:37:28.946686 19076 leader_election.cc:302] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a9e72891ec410ba901705dfe969cc8; no voters:
I20260927 16:37:28.947067 19076 leader_election.cc:288] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:28.947196 19081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:28.947424 19081 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [term 1 LEADER]: Becoming Leader. State: Replica: a8a9e72891ec410ba901705dfe969cc8, State: Running, Role: LEADER
I20260927 16:37:28.947739 19081 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } }
I20260927 16:37:28.948400 19076 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:28.948961 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a9e72891ec410ba901705dfe969cc8. Latest consensus state: current_term: 1 leader_uuid: "a8a9e72891ec410ba901705dfe969cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } } }
I20260927 16:37:28.949074 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:28.949805 19083 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a9e72891ec410ba901705dfe969cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a9e72891ec410ba901705dfe969cc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41043 } } }
I20260927 16:37:28.949893 19083 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:28.950445 19085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:28.953694 19085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:28.959179 19085 catalog_manager.cc:1347] Generated new cluster ID: 14bf4255437c47aba5d1b8686751fb9a
I20260927 16:37:28.959254 19085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:28.976481 19085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:28.977417 19085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:28.985045 19085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: Generated new TSK 0
I20260927 16:37:28.985801 19085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:28.997400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--builtin_ntp_servers=127.0.98.20:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:29.114082 19100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:29.114408 19100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:29.114446 19100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:29.118705 19100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:29.118769 19100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:29.118881 19100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:29.123833 19100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:29.124943 19100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:29.126361 19100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:29.135741 19105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:29.135890 19100 server_base.cc:1034] running on GCE node
W20260927 16:37:29.135809 19106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:29.135839 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:29.136395 19100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:29.137017 19100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:29.138211 19100 hybrid_clock.cc:648] HybridClock initialized: now 1790527049138191 us; error 29 us; skew 500 ppm
I20260927 16:37:29.140053 19100 webserver.cc:460] Webserver started at http://127.0.98.1:41493/ using document root <none> and password file <none>
I20260927 16:37:29.140632 19100 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:29.140693 19100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:29.140940 19100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:29.143079 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "5dfd4ba8ac9348b6a72e755e25528a71"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "23425d40734c54e677a30a1e8e0d0837"
server_key_iv: "5810260fcba058e357762b0bfa4ad27f"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.143791 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5dfd4ba8ac9348b6a72e755e25528a71"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "23425d40734c54e677a30a1e8e0d0837"
server_key_iv: "5810260fcba058e357762b0bfa4ad27f"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.147477 19100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:29.150043 19114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.151424 19100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:37:29.151556 19100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "5dfd4ba8ac9348b6a72e755e25528a71"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "23425d40734c54e677a30a1e8e0d0837"
server_key_iv: "5810260fcba058e357762b0bfa4ad27f"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.151678 19100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:29.168133 19100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:29.168853 19100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:29.169057 19100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:29.169648 19100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:29.170804 19100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:29.170873 19100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.170915 19100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:29.170972 19100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.179939 19100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42159
I20260927 16:37:29.179956 19227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42159 every 8 connection(s)
I20260927 16:37:29.180891 19100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:37:29.184388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19100
I20260927 16:37:29.184482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:29.184754 392 external_mini_cluster.cc:1428] Setting key 0968776a59667ecc5d892034a427221d
I20260927 16:37:29.186774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--builtin_ntp_servers=127.0.98.20:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:29.190801 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.182776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40473 (local address 127.0.98.62:41043)
0927 16:37:29.182909 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.182913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.183994 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 16:37:29.185639 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.185642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.185703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.185790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:29.186880 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.187467 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.188658 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.188896 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.189012 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.190235 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:29.190239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:29.190242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:29.190257 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:29.190285 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:29.190293 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:29.190375 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.190547 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.190645 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:29.191814 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.183181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41043 (local address 127.0.98.1:40473)
0927 16:37:29.183840 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.183883 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.185409 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.185982 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.185990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.186122 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:29.186727 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.186740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.187609 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.187613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.188517 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.188535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.189056 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.189060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.189244 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.189924 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:29.189925 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:29.189972 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:29.190046 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:29.190376 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:29.190386 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:29.190394 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.190557 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.190887 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260927 16:37:29.193300 19228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41043
I20260927 16:37:29.193612 19228 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:29.194435 19228 heartbeater.cc:511] Master 127.0.98.62:41043 requested a full tablet report, sending...
I20260927 16:37:29.196390 19040 ts_manager.cc:194] Registered new tserver with Master: 5dfd4ba8ac9348b6a72e755e25528a71 (127.0.98.1:42159)
I20260927 16:37:29.198247 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40473
W20260927 16:37:29.298969 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:29.299335 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:29.299417 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:29.304013 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:29.304067 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:29.304121 19231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:29.308738 19231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:29.309757 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:29.311177 19231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:29.321404 19231 server_base.cc:1034] running on GCE node
W20260927 16:37:29.321389 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:29.321354 19236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:29.321354 19237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:29.322023 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:29.322758 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:29.323930 19231 hybrid_clock.cc:648] HybridClock initialized: now 1790527049323909 us; error 34 us; skew 500 ppm
I20260927 16:37:29.325773 19231 webserver.cc:460] Webserver started at http://127.0.98.2:34255/ using document root <none> and password file <none>
I20260927 16:37:29.326408 19231 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:29.326474 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:29.326651 19231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:29.328672 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "800bfbabd6c44d62843db18f62750447"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "5937fe29af18c811eaffb705b43b9c65"
server_key_iv: "8b124fcb148d6777e43b8397e3db4acc"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.329192 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "800bfbabd6c44d62843db18f62750447"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "5937fe29af18c811eaffb705b43b9c65"
server_key_iv: "8b124fcb148d6777e43b8397e3db4acc"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.332881 19231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:37:29.335379 19245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.336696 19231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:29.336795 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "800bfbabd6c44d62843db18f62750447"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "5937fe29af18c811eaffb705b43b9c65"
server_key_iv: "8b124fcb148d6777e43b8397e3db4acc"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.336894 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:29.355075 19231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:29.355882 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:29.356094 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:29.356791 19231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:29.357774 19231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:29.357822 19231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.357894 19231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:29.357932 19231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.366770 19231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40861
I20260927 16:37:29.366797 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40861 every 8 connection(s)
I20260927 16:37:29.367682 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:29.373250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19231
I20260927 16:37:29.373378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:29.373667 392 external_mini_cluster.cc:1428] Setting key 731dd4038532e23bc0d59d2f9e11b64f
I20260927 16:37:29.375542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--builtin_ntp_servers=127.0.98.20:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:29.377134 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.369437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60093 (local address 127.0.98.62:41043)
0927 16:37:29.369592 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.369595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.370689 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 16:37:29.372073 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.372077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.372136 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.372240 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:29.373115 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.373788 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.375126 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.375359 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.375470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.376642 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:29.376645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:29.376648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:29.376667 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:29.376685 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:29.376693 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:29.376772 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.376936 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.376997 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:37:29.377938 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.369846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41043 (local address 127.0.98.2:60093)
0927 16:37:29.370539 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.370581 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.371851 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.372412 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.372420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.372500 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:29.372959 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.372970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.373908 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.373914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.375002 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.375022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.375503 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.375507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.375715 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.376375 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:29.376376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:29.376415 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:29.376496 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:29.376773 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:29.376784 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:29.376792 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.376921 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.377193 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260927 16:37:29.379114 19359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41043
I20260927 16:37:29.379367 19359 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:29.380012 19359 heartbeater.cc:511] Master 127.0.98.62:41043 requested a full tablet report, sending...
I20260927 16:37:29.381174 19040 ts_manager.cc:194] Registered new tserver with Master: 800bfbabd6c44d62843db18f62750447 (127.0.98.2:40861)
I20260927 16:37:29.382040 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60093
W20260927 16:37:29.489778 19362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:29.490028 19362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:29.490075 19362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:29.494051 19362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:29.494110 19362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:29.494217 19362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:29.498924 19362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:29.499922 19362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:29.501353 19362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:29.511090 19368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:29.511111 19370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:29.511101 19367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:29.511304 19362 server_base.cc:1034] running on GCE node
I20260927 16:37:29.511818 19362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:29.512526 19362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:29.513729 19362 hybrid_clock.cc:648] HybridClock initialized: now 1790527049513712 us; error 34 us; skew 500 ppm
I20260927 16:37:29.515791 19362 webserver.cc:460] Webserver started at http://127.0.98.3:44247/ using document root <none> and password file <none>
I20260927 16:37:29.516423 19362 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:29.516520 19362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:29.516764 19362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:29.519134 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "ffcd5ecaa97147f18c81b79e34fb4221"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "8414754dd5901aee84dd65ec96d5fbff"
server_key_iv: "9d1bd10cbaa0867440605cf46fe97e10"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.519722 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ffcd5ecaa97147f18c81b79e34fb4221"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "8414754dd5901aee84dd65ec96d5fbff"
server_key_iv: "9d1bd10cbaa0867440605cf46fe97e10"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.523427 19362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:29.525766 19376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.527025 19362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:29.527184 19362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "ffcd5ecaa97147f18c81b79e34fb4221"
format_stamp: "Formatted at 2026-09-27 16:37:29 on dist-test-slave-t27c"
server_key: "8414754dd5901aee84dd65ec96d5fbff"
server_key_iv: "9d1bd10cbaa0867440605cf46fe97e10"
server_key_version: "encryptionkey@0"
I20260927 16:37:29.527295 19362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:29.536237 19362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:29.536844 19362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:29.537021 19362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:29.537562 19362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:29.538496 19362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:29.538586 19362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.538658 19362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:29.538712 19362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:29.547338 19362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36667
I20260927 16:37:29.547359 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36667 every 8 connection(s)
I20260927 16:37:29.548220 19362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:29.551744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19362
I20260927 16:37:29.551839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:29.552121 392 external_mini_cluster.cc:1428] Setting key ae3e5f67ffba30c4aef74fc6bcffd1d5
I20260927 16:37:29.557928 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.550094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53275 (local address 127.0.98.62:41043)
0927 16:37:29.550288 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.550291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.551469 (+ 1178us) server_negotiation.cc:407] Connection header received
0927 16:37:29.553179 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.553182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.553240 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.553328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:29.554368 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.554958 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.556197 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.556478 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.556566 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.557559 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:29.557562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:29.557564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:29.557575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:29.557593 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:29.557600 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:29.557657 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.557763 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.557812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:37:29.558890 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.550590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41043 (local address 127.0.98.3:53275)
0927 16:37:29.551302 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.551343 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.552915 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.553517 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.553528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.553637 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:29.554155 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.554219 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.555077 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.555079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.556091 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.556100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.556624 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.556626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.556766 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.557307 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:29.557309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:29.557350 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:29.557415 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:29.557656 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:29.557660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:29.557663 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.557747 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.557958 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":126,"threads_started":1}
I20260927 16:37:29.560034 19490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41043
I20260927 16:37:29.560264 19490 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:29.560774 19490 heartbeater.cc:511] Master 127.0.98.62:41043 requested a full tablet report, sending...
I20260927 16:37:29.561758 19040 ts_manager.cc:194] Registered new tserver with Master: ffcd5ecaa97147f18c81b79e34fb4221 (127.0.98.3:36667)
I20260927 16:37:29.562372 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53275
I20260927 16:37:29.565996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:29.571269 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.567902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.98.62:41043)
0927 16:37:29.568023 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.568027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.568225 (+ 198us) server_negotiation.cc:407] Connection header received
0927 16:37:29.568315 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.568317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.568352 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.568416 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:29.568652 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.569127 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.569867 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.570063 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.570134 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.570904 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:29.570907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:29.570909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:29.570920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:29.570937 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:29.570944 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:29.571041 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.571138 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.571165 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:29.574023 19040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:29.577052 19040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:29.591022 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.585820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.0.98.3:36667)
0927 16:37:29.586448 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.586460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.586484 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:37:29.586572 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.586583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.586773 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.586924 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:29.587284 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.588361 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.589598 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590301 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590421 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.590634 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.590730 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.590835 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":91,"threads_started":1}
I20260927 16:37:29.591370 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.585008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.2:40861)
0927 16:37:29.586156 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.586194 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.586224 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:37:29.586304 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.586311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.586562 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.586716 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:29.587024 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.587889 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.589564 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590328 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590432 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.590633 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.591086 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.591202 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":136,"threads_started":1}
I20260927 16:37:29.591439 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.585367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.98.1:42159)
0927 16:37:29.586094 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.586110 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.586463 (+ 353us) server_negotiation.cc:407] Connection header received
0927 16:37:29.586575 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.586580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.586752 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.586882 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:29.587126 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.588289 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.589854 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590620 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590718 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.590888 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.591190 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.591284 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":108,"threads_started":1}
I20260927 16:37:29.591012 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.585623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36667 (local address 127.0.0.1:37958)
0927 16:37:29.586164 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.586206 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.586289 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.587074 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.587079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.587098 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:29.587167 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.587172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.588448 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.588453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.589476 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.589485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590424 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.590538 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.590576 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.590702 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.590860 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260927 16:37:29.591414 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.584892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40861 (local address 127.0.0.1:52218)
0927 16:37:29.585697 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.585745 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.585865 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.586715 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.586721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.586741 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:29.586893 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.586900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.588008 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.588014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.589425 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.589435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590699 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.590807 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.590829 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.590901 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.590974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260927 16:37:29.591869 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.584968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42159 (local address 127.0.0.1:52668)
0927 16:37:29.586352 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.586367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.586464 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.586884 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.586887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.586904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:29.587002 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.587008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.588413 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.588416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.589734 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.589742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.590945 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.590948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.591040 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.591062 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.591624 (+ 562us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.591712 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":25,"thread_start_us":87,"threads_started":1}
I20260927 16:37:29.594527 19424 tablet_service.cc:1467] Processing CreateTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b (DEFAULT_TABLE table=test-table [id=e0f9475bee214beb8e2d8c51356ce40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:29.594611 19293 tablet_service.cc:1467] Processing CreateTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b (DEFAULT_TABLE table=test-table [id=e0f9475bee214beb8e2d8c51356ce40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:29.595078 19162 tablet_service.cc:1467] Processing CreateTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b (DEFAULT_TABLE table=test-table [id=e0f9475bee214beb8e2d8c51356ce40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:29.595609 19424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f6cd8f51c4441590dc9fc50e92dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:29.595659 19293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f6cd8f51c4441590dc9fc50e92dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:29.595988 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f6cd8f51c4441590dc9fc50e92dd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:29.601843 19507 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Bootstrap starting.
I20260927 16:37:29.603447 19509 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Bootstrap starting.
I20260927 16:37:29.603447 19508 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Bootstrap starting.
I20260927 16:37:29.604794 19507 tablet_bootstrap.cc:654] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:29.605504 19507 log.cc:824] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:29.605571 19509 tablet_bootstrap.cc:654] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:29.605574 19508 tablet_bootstrap.cc:654] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:29.606431 19509 log.cc:824] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:29.606431 19508 log.cc:824] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:29.607607 19507 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: No bootstrap required, opened a new log
I20260927 16:37:29.607812 19507 ts_tablet_manager.cc:1397] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:37:29.608341 19509 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: No bootstrap required, opened a new log
I20260927 16:37:29.608530 19509 ts_tablet_manager.cc:1397] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:37:29.608681 19508 tablet_bootstrap.cc:492] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: No bootstrap required, opened a new log
I20260927 16:37:29.608876 19508 ts_tablet_manager.cc:1397] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:37:29.612917 19507 raft_consensus.cc:348] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.612946 19509 raft_consensus.cc:348] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.612969 19508 raft_consensus.cc:348] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.613178 19507 raft_consensus.cc:374] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:29.613204 19508 raft_consensus.cc:374] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:29.613204 19509 raft_consensus.cc:374] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:29.613240 19507 raft_consensus.cc:729] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcd5ecaa97147f18c81b79e34fb4221, State: Initialized, Role: FOLLOWER
I20260927 16:37:29.613255 19509 raft_consensus.cc:729] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfd4ba8ac9348b6a72e755e25528a71, State: Initialized, Role: FOLLOWER
I20260927 16:37:29.613255 19508 raft_consensus.cc:729] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800bfbabd6c44d62843db18f62750447, State: Initialized, Role: FOLLOWER
I20260927 16:37:29.613713 19507 consensus_queue.cc:260] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.613715 19509 consensus_queue.cc:260] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.613715 19508 consensus_queue.cc:260] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.614444 19359 heartbeater.cc:503] Master 127.0.98.62:41043 was elected leader, sending a full tablet report...
I20260927 16:37:29.614699 19228 heartbeater.cc:503] Master 127.0.98.62:41043 was elected leader, sending a full tablet report...
I20260927 16:37:29.614804 19507 ts_tablet_manager.cc:1428] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260927 16:37:29.614804 19509 ts_tablet_manager.cc:1428] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:37:29.615123 19490 heartbeater.cc:503] Master 127.0.98.62:41043 was elected leader, sending a full tablet report...
I20260927 16:37:29.615754 19508 ts_tablet_manager.cc:1428] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20260927 16:37:29.618548 19360 tablet.cc:2369] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:37:29.682061 19229 tablet.cc:2369] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:37:29.737411 19513 raft_consensus.cc:482] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:29.737640 19513 raft_consensus.cc:504] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.738909 19513 leader_election.cc:288] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dfd4ba8ac9348b6a72e755e25528a71 (127.0.98.1:42159), ffcd5ecaa97147f18c81b79e34fb4221 (127.0.98.3:36667)
I20260927 16:37:29.743106 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.739120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42159 (local address 127.0.98.2:42987)
0927 16:37:29.739329 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.739345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.739456 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.739765 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.739768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.739790 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:29.739916 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.739923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.740785 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.740789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.741937 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.741946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.742670 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.742778 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.742805 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.742883 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.742938 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":18}
I20260927 16:37:29.743124 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.739178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42987 (local address 127.0.98.1:42159)
0927 16:37:29.739309 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.739315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.739422 (+ 107us) server_negotiation.cc:407] Connection header received
0927 16:37:29.739574 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.739577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.739632 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.739729 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:29.740033 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.740668 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742063 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.742556 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742638 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.742714 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.742916 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.743000 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:29.743172 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.739128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36667 (local address 127.0.98.2:33737)
0927 16:37:29.739659 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.739675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:29.739770 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:29.740025 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:29.740028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:29.740079 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:29.740191 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.740197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.741103 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.741106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.741951 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:29.741960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.742751 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:29.742845 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.742899 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 16:37:29.742973 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 16:37:29.743032 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":182,"thread_start_us":136,"threads_started":1}
I20260927 16:37:29.743193 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.739225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33737 (local address 127.0.98.3:36667)
0927 16:37:29.739358 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.739365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.739761 (+ 396us) server_negotiation.cc:407] Connection header received
0927 16:37:29.739864 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.739867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.739917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.739999 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:29.740319 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.740988 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742101 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.742624 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.742727 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.742787 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.743008 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.743073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:37:29.743746 19182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f6cd8f51c4441590dc9fc50e92dd3b" candidate_uuid: "800bfbabd6c44d62843db18f62750447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" is_pre_election: true
I20260927 16:37:29.743820 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f6cd8f51c4441590dc9fc50e92dd3b" candidate_uuid: "800bfbabd6c44d62843db18f62750447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" is_pre_election: true
I20260927 16:37:29.744072 19182 raft_consensus.cc:2454] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800bfbabd6c44d62843db18f62750447 in term 0.
I20260927 16:37:29.744102 19444 raft_consensus.cc:2454] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800bfbabd6c44d62843db18f62750447 in term 0.
I20260927 16:37:29.744577 19249 leader_election.cc:302] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5dfd4ba8ac9348b6a72e755e25528a71, 800bfbabd6c44d62843db18f62750447, ffcd5ecaa97147f18c81b79e34fb4221; no voters:
I20260927 16:37:29.744854 19513 raft_consensus.cc:2781] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:29.744925 19513 raft_consensus.cc:482] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:29.744953 19513 raft_consensus.cc:3037] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:29.745981 19513 raft_consensus.cc:504] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.746367 19513 leader_election.cc:288] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [CANDIDATE]: Term 1 election: Requested vote from peers 5dfd4ba8ac9348b6a72e755e25528a71 (127.0.98.1:42159), ffcd5ecaa97147f18c81b79e34fb4221 (127.0.98.3:36667)
I20260927 16:37:29.746755 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f6cd8f51c4441590dc9fc50e92dd3b" candidate_uuid: "800bfbabd6c44d62843db18f62750447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcd5ecaa97147f18c81b79e34fb4221"
I20260927 16:37:29.746893 19444 raft_consensus.cc:3037] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:29.747879 19444 raft_consensus.cc:2454] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800bfbabd6c44d62843db18f62750447 in term 1.
I20260927 16:37:29.748142 19182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f6cd8f51c4441590dc9fc50e92dd3b" candidate_uuid: "800bfbabd6c44d62843db18f62750447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dfd4ba8ac9348b6a72e755e25528a71"
I20260927 16:37:29.748201 19249 leader_election.cc:302] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800bfbabd6c44d62843db18f62750447, ffcd5ecaa97147f18c81b79e34fb4221; no voters:
I20260927 16:37:29.748307 19182 raft_consensus.cc:3037] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:29.748383 19513 raft_consensus.cc:2781] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:29.748669 19513 raft_consensus.cc:686] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 1 LEADER]: Becoming Leader. State: Replica: 800bfbabd6c44d62843db18f62750447, State: Running, Role: LEADER
I20260927 16:37:29.749128 19513 consensus_queue.cc:237] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } }
I20260927 16:37:29.750767 19182 raft_consensus.cc:2454] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800bfbabd6c44d62843db18f62750447 in term 1.
I20260927 16:37:29.752811 19040 catalog_manager.cc:5495] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 reported cstate change: term changed from 0 to 1, leader changed from <none> to 800bfbabd6c44d62843db18f62750447 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "800bfbabd6c44d62843db18f62750447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bfbabd6c44d62843db18f62750447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 } health_report { overall_health: UNKNOWN } } }
I20260927 16:37:29.782917 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.779599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.98.2:40861)
0927 16:37:29.779734 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:29.779737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:29.779752 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:37:29.779825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:29.779827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:29.779882 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:29.779972 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:29.780180 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.780666 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.781443 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:29.781646 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:29.781719 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:29.781965 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:29.782431 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:29.782545 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:29.782680 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 16:37:29.782711 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:37:29.788802 19444 raft_consensus.cc:1261] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 1 FOLLOWER]: Refusing update from remote peer 800bfbabd6c44d62843db18f62750447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:29.788802 19182 raft_consensus.cc:1261] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71 [term 1 FOLLOWER]: Refusing update from remote peer 800bfbabd6c44d62843db18f62750447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:29.789485 19513 consensus_queue.cc:1035] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dfd4ba8ac9348b6a72e755e25528a71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:29.789700 19517 consensus_queue.cc:1035] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffcd5ecaa97147f18c81b79e34fb4221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:29.798588 19522 mvcc.cc:205] Tried to move back new op lower bound from 7333998795927744512 to 7333998795776081920. Current Snapshot: MvccSnapshot[applied={T|T < 7333998795927744512}]
I20260927 16:37:29.806272 19039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:29.806453 19039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:29.808759 19039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8a9e72891ec410ba901705dfe969cc8: Sending DeleteTablet for 3 replicas of tablet 67f6cd8f51c4441590dc9fc50e92dd3b
I20260927 16:37:29.809551 19293 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:29 UTC) from {username='slave'} at 127.0.0.1:52218
I20260927 16:37:29.809571 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:29 UTC) from {username='slave'} at 127.0.0.1:52668
I20260927 16:37:29.809718 19424 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f6cd8f51c4441590dc9fc50e92dd3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:29 UTC) from {username='slave'} at 127.0.0.1:37958
I20260927 16:37:29.810036 19529 tablet_replica.cc:330] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: stopping tablet replica
I20260927 16:37:29.810353 19529 raft_consensus.cc:2229] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:37:29.810367 19531 tablet_replica.cc:330] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: stopping tablet replica
I20260927 16:37:29.810515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19100
I20260927 16:37:29.810683 19530 tablet_replica.cc:330] T 67f6cd8f51c4441590dc9fc50e92dd3b P 5dfd4ba8ac9348b6a72e755e25528a71: stopping tablet replica
I20260927 16:37:29.810763 19529 raft_consensus.cc:2258] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:29.811281 19531 raft_consensus.cc:2229] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:29.811794 19531 raft_consensus.cc:2258] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:29.812597 19529 ts_tablet_manager.cc:1905] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:29.813280 19531 ts_tablet_manager.cc:1905] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:29.815609 19529 ts_tablet_manager.cc:1918] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:29.815709 19529 log.cc:1196] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/67f6cd8f51c4441590dc9fc50e92dd3b
I20260927 16:37:29.816041 19529 ts_tablet_manager.cc:1939] T 67f6cd8f51c4441590dc9fc50e92dd3b P 800bfbabd6c44d62843db18f62750447: Deleting consensus metadata
I20260927 16:37:29.816242 19531 ts_tablet_manager.cc:1918] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:29.816380 19531 log.cc:1196] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/67f6cd8f51c4441590dc9fc50e92dd3b
I20260927 16:37:29.816732 19531 ts_tablet_manager.cc:1939] T 67f6cd8f51c4441590dc9fc50e92dd3b P ffcd5ecaa97147f18c81b79e34fb4221: Deleting consensus metadata
I20260927 16:37:29.817278 19027 catalog_manager.cc:4841] TS 800bfbabd6c44d62843db18f62750447 (127.0.98.2:40861): tablet 67f6cd8f51c4441590dc9fc50e92dd3b (table test-table [id=e0f9475bee214beb8e2d8c51356ce40e]) successfully deleted
I20260927 16:37:29.819857 19027 catalog_manager.cc:4841] TS ffcd5ecaa97147f18c81b79e34fb4221 (127.0.98.3:36667): tablet 67f6cd8f51c4441590dc9fc50e92dd3b (table test-table [id=e0f9475bee214beb8e2d8c51356ce40e]) successfully deleted
W20260927 16:37:29.820343 19025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42159 (error 108)
I20260927 16:37:29.820544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19231
I20260927 16:37:29.821558 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:29.821184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42159 (local address 127.0.0.1:52672)
0927 16:37:29.821378 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:29.821456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 16:37:29.822041 19025 catalog_manager.cc:4584] TS 5dfd4ba8ac9348b6a72e755e25528a71 (127.0.98.1:42159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67f6cd8f51c4441590dc9fc50e92dd3b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42159: connect: Connection refused (error 111)
I20260927 16:37:29.826508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19362
I20260927 16:37:29.831991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19009
2026-09-27T16:37:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1111 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:29 dist-test-slave-t27c krb5kdc[19535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:29 dist-test-slave-t27c krb5kdc[19535](info): set up 2 sockets
Sep 27 16:37:29 dist-test-slave-t27c krb5kdc[19535](info): commencing operation
krb5kdc: starting...
W20260927 16:37:30.870263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:30 dist-test-slave-t27c krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:30.995395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:31.109992 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:31.110365 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:31.110462 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:31.114692 19548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:31.114807 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:31.114842 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:31.114886 19548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:31.114921 19548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:31.120532 19548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:31.121732 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:31.122841 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:31.131394 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:31.131407 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:31.131369 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.131446 19548 server_base.cc:1034] running on GCE node
I20260927 16:37:31.132156 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:31.133354 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:31.134531 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790527051134515 us; error 29 us; skew 500 ppm
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:31.137212 19548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:31.138057 19548 webserver.cc:460] Webserver started at http://127.0.98.62:35153/ using document root <none> and password file <none>
I20260927 16:37:31.138659 19548 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:31.138721 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:31.138883 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:31.140877 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b8539fb61f764e2ca2336461c167cb62"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "3a85bd51966d873f6df18d7335761ce7"
server_key_iv: "47dd5c7a550f3dab968be3055773c584"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.141453 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8539fb61f764e2ca2336461c167cb62"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "3a85bd51966d873f6df18d7335761ce7"
server_key_iv: "47dd5c7a550f3dab968be3055773c584"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.145279 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:37:31.147739 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.148965 19548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:31.149123 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b8539fb61f764e2ca2336461c167cb62"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "3a85bd51966d873f6df18d7335761ce7"
server_key_iv: "47dd5c7a550f3dab968be3055773c584"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.149222 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:31.171830 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:31.174829 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:31.175032 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:31.182206 19548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35499
I20260927 16:37:31.182266 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35499 every 8 connection(s)
I20260927 16:37:31.183212 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:31.185988 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:31.191458 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: Bootstrap starting.
I20260927 16:37:31.192114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19548
I20260927 16:37:31.192281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:31.192543 392 external_mini_cluster.cc:1428] Setting key 10af977bbc47ad1547dba7591f5c36cd
I20260927 16:37:31.194438 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:31.195216 19616 log.cc:824] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:31.197168 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: No bootstrap required, opened a new log
I20260927 16:37:31.200742 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } }
I20260927 16:37:31.201004 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:31.201095 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8539fb61f764e2ca2336461c167cb62, State: Initialized, Role: FOLLOWER
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:31.201642 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } }
I20260927 16:37:31.201830 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:31.201907 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:31.202014 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:31.203002 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } }
I20260927 16:37:31.203356 19616 leader_election.cc:302] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8539fb61f764e2ca2336461c167cb62; no voters:
I20260927 16:37:31.203682 19616 leader_election.cc:288] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:31.203801 19621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:31.204001 19621 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [term 1 LEADER]: Becoming Leader. State: Replica: b8539fb61f764e2ca2336461c167cb62, State: Running, Role: LEADER
I20260927 16:37:31.204334 19621 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } }
I20260927 16:37:31.205024 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:31.205714 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8539fb61f764e2ca2336461c167cb62. Latest consensus state: current_term: 1 leader_uuid: "b8539fb61f764e2ca2336461c167cb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } } }
I20260927 16:37:31.205850 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:31.206708 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8539fb61f764e2ca2336461c167cb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8539fb61f764e2ca2336461c167cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35499 } } }
I20260927 16:37:31.206903 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:31.207649 19628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:31.208415 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.193854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.0.98.62:35499)
0927 16:37:31.194382 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.194399 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.194437 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:37:31.195164 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.195183 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.195573 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.195919 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:31.196740 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.197802 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.198760 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.199256 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.199435 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.202201 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:31.202208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:31.202225 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:31.202262 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:31.204401 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.204926 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.204931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.204937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.205081 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.205609 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.205612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.205614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.206153 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:31.206389 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.207155 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.207485 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260927 16:37:31.210913 19628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:31.216306 19628 catalog_manager.cc:1347] Generated new cluster ID: 54a2a84cc77b4946b4afb20822237911
I20260927 16:37:31.216373 19628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:31.225217 19628 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:31.226248 19628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:31.242846 19628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: Generated new TSK 0
I20260927 16:37:31.243688 19628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:37:31.275511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:31.395578 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:31.395929 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:31.396008 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:31.400220 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:31.400316 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:31.400410 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:31.406093 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:31.407315 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:31.408345 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:31.418011 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:31.418017 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.418053 19642 server_base.cc:1034] running on GCE node
W20260927 16:37:31.418076 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.418629 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:31.419318 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:31.420497 19642 hybrid_clock.cc:648] HybridClock initialized: now 1790527051420481 us; error 30 us; skew 500 ppm
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:31.423353 19642 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:37:31.424192 19642 webserver.cc:460] Webserver started at http://127.0.98.1:36927/ using document root <none> and password file <none>
I20260927 16:37:31.424765 19642 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:31.424854 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:31.425082 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:31.427162 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd6a420d33e545d8b407c92adca83131"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "7aad022fe381b5078a1461df4eee7907"
server_key_iv: "566e8aa1a0c7e4abd67da1c105c9cf76"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.427732 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd6a420d33e545d8b407c92adca83131"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "7aad022fe381b5078a1461df4eee7907"
server_key_iv: "566e8aa1a0c7e4abd67da1c105c9cf76"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.431476 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:31.434239 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.435535 19642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:37:31.435693 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "dd6a420d33e545d8b407c92adca83131"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "7aad022fe381b5078a1461df4eee7907"
server_key_iv: "566e8aa1a0c7e4abd67da1c105c9cf76"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.435828 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:31.448603 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:31.451529 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:31.451733 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:31.452307 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:31.453258 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:31.453328 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.453395 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:31.453444 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.462039 19642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43253
I20260927 16:37:31.462066 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43253 every 8 connection(s)
I20260927 16:37:31.462965 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:31.473119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19642
I20260927 16:37:31.473287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:31.473582 392 external_mini_cluster.cc:1428] Setting key 50872805c9ab9f2da03e4bf564c4532d
I20260927 16:37:31.477294 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.464783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34931 (local address 127.0.98.62:35499)
0927 16:37:31.464919 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.464922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.465801 (+ 879us) server_negotiation.cc:407] Connection header received
0927 16:37:31.467128 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.467131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.467183 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.467260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:31.468543 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.469072 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.470154 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.470435 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.470519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.473171 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:31.473174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:31.473176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:31.473202 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:31.474979 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.475510 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.475514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.475516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.475605 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.476119 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.476123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.476125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.476337 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:31.476452 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.477000 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.477133 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:37:31.478333 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.465193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35499 (local address 127.0.98.1:34931)
0927 16:37:31.465640 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.465685 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.466925 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.467452 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.467459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.467829 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:31.468420 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.468434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.469178 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.469180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.470040 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.470047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.470601 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.470605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.470762 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.471352 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:31.471369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:31.472966 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:31.475084 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.475091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.475109 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.475399 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.475702 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.475706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.475708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.475862 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.476452 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:31.476461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:31.476748 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.476961 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.477369 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
I20260927 16:37:31.479687 19771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35499
I20260927 16:37:31.479938 19771 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:31.480438 19771 heartbeater.cc:511] Master 127.0.98.62:35499 requested a full tablet report, sending...
I20260927 16:37:31.482155 19580 ts_manager.cc:194] Registered new tserver with Master: dd6a420d33e545d8b407c92adca83131 (127.0.98.1:43253)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:37:31.483850 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:37:31.501212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:31.611261 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:31.611531 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:31.611575 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:31.615633 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:31.615691 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:31.615787 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:31.621023 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:31.622064 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:31.623086 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:31.632404 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.632463 19776 server_base.cc:1034] running on GCE node
W20260927 16:37:31.632498 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:31.632416 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.633144 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:31.633881 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:31.635077 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790527051635034 us; error 57 us; skew 500 ppm
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:31.637856 19776 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:37:31.638816 19776 webserver.cc:460] Webserver started at http://127.0.98.2:40863/ using document root <none> and password file <none>
I20260927 16:37:31.639402 19776 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:31.639492 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:31.639732 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:31.641783 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "c02055c6bfba4309a758d1aaba032728"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "35e15767c82db6ded85cb7707c3a39d0"
server_key_iv: "28a61a700a948e0b542edb1c64dcdf32"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.642405 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c02055c6bfba4309a758d1aaba032728"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "35e15767c82db6ded85cb7707c3a39d0"
server_key_iv: "28a61a700a948e0b542edb1c64dcdf32"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.646044 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:37:31.648630 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.649816 19776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:31.649950 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "c02055c6bfba4309a758d1aaba032728"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "35e15767c82db6ded85cb7707c3a39d0"
server_key_iv: "28a61a700a948e0b542edb1c64dcdf32"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.650059 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:31.662436 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:31.665072 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:31.665266 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:31.665812 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:31.666689 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:31.666759 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.666827 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:31.666877 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.675958 19776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39683
I20260927 16:37:31.675978 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39683 every 8 connection(s)
I20260927 16:37:31.676937 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:31.687039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19776
I20260927 16:37:31.687181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:31.687467 392 external_mini_cluster.cc:1428] Setting key 1fcb7d4de2079cf4f2769d5a561013fa
I20260927 16:37:31.690779 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.678949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41785 (local address 127.0.98.62:35499)
0927 16:37:31.679095 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.679099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.679942 (+ 843us) server_negotiation.cc:407] Connection header received
0927 16:37:31.681138 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.681141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.681193 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.681266 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:31.682484 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.682994 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.684080 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.684297 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.684376 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.686765 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:31.686768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:31.686770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:31.686810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:31.688408 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.689082 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.689085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.689086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.689190 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.689525 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.689531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.689534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.689780 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:31.689882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.690401 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.690535 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:37:31.691511 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.679351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35499 (local address 127.0.98.2:41785)
0927 16:37:31.679778 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.679820 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.680944 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.681429 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.681436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.681793 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:31.682351 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.682366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.683096 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.683099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.683966 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.683974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.684426 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.684429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.684566 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.685122 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:31.685138 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:31.686601 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:31.688516 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.688521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.688537 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.688885 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.689285 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.689288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.689289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.689425 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.689880 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:31.689884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:31.690116 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.690329 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.690606 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260927 16:37:31.692755 19905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35499
I20260927 16:37:31.693069 19905 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:31.693600 19905 heartbeater.cc:511] Master 127.0.98.62:35499 requested a full tablet report, sending...
I20260927 16:37:31.694708 19580 ts_manager.cc:194] Registered new tserver with Master: c02055c6bfba4309a758d1aaba032728 (127.0.98.2:39683)
I20260927 16:37:31.695282 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41785
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:37:31.717528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:31.828869 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:31.829165 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:31.829234 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:31.833377 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:31.833477 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:31.833572 19910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:31.839171 19910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:31.840410 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:31.841456 19910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:31.850803 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.850862 19910 server_base.cc:1034] running on GCE node
W20260927 16:37:31.851006 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:31.850951 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:31.851460 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:31.852120 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:31.853310 19910 hybrid_clock.cc:648] HybridClock initialized: now 1790527051853293 us; error 29 us; skew 500 ppm
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:31.856139 19910 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:37:31.856957 19910 webserver.cc:460] Webserver started at http://127.0.98.3:43573/ using document root <none> and password file <none>
I20260927 16:37:31.857453 19910 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:31.857512 19910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:31.857668 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:31.859702 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c22e236e07145d3a25efed6d9941675"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "40700932b37458f16d9417057a1195d9"
server_key_iv: "8adc99123981f5f3d3973a582965026d"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.860200 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c22e236e07145d3a25efed6d9941675"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "40700932b37458f16d9417057a1195d9"
server_key_iv: "8adc99123981f5f3d3973a582965026d"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.863783 19910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:37:31.866016 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.867199 19910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:37:31.867291 19910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "0c22e236e07145d3a25efed6d9941675"
format_stamp: "Formatted at 2026-09-27 16:37:31 on dist-test-slave-t27c"
server_key: "40700932b37458f16d9417057a1195d9"
server_key_iv: "8adc99123981f5f3d3973a582965026d"
server_key_version: "encryptionkey@0"
I20260927 16:37:31.867369 19910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:31.878398 19910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:31.881157 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:31.881305 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:31.881830 19910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:31.882683 19910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:31.882730 19910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.882771 19910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:31.882789 19910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:31.891867 19910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37687
I20260927 16:37:31.891885 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37687 every 8 connection(s)
I20260927 16:37:31.892769 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:31.893770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 19910
I20260927 16:37:31.893911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:31.894312 392 external_mini_cluster.cc:1428] Setting key 6a5a2318995e72db47be3d2f503bbff3
Sep 27 16:37:31 dist-test-slave-t27c krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:31.905834 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.894758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34013 (local address 127.0.98.62:35499)
0927 16:37:31.894900 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.894904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.895894 (+ 990us) server_negotiation.cc:407] Connection header received
0927 16:37:31.897153 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.897156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.897200 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.897268 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:31.898450 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.898979 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.900031 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.900236 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.900309 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.902597 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:31.902600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:31.902602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:31.902635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:31.904073 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.904494 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.904496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.904498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.904567 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.904841 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.904843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.904844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.905066 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:31.905145 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.905598 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.905695 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:31.906852 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.895249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35499 (local address 127.0.98.3:34013)
0927 16:37:31.895744 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.895796 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.896970 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.897429 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.897435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.897784 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:31.898334 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.898349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.899079 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.899082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.899930 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.899938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.900352 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.900354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.900485 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.901021 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:37:31.901036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:31.902453 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:31.904166 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.904171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.904181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.904410 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.904642 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:37:31.904644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:37:31.904645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:37:31.904766 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:37:31.905158 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:31.905163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:31.905374 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.905571 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.905855 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260927 16:37:31.907931 20039 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35499
I20260927 16:37:31.908181 20039 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:31.908613 20039 heartbeater.cc:511] Master 127.0.98.62:35499 requested a full tablet report, sending...
I20260927 16:37:31.909525 19580 ts_manager.cc:194] Registered new tserver with Master: 0c22e236e07145d3a25efed6d9941675 (127.0.98.3:37687)
I20260927 16:37:31.910032 19580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34013
I20260927 16:37:31.920096 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:31.929293 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.922224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.98.62:35499)
0927 16:37:31.922367 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.922372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.922554 (+ 182us) server_negotiation.cc:407] Connection header received
0927 16:37:31.922650 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.922652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.922694 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.922760 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:31.923442 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.923948 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.924759 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.924953 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.925026 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.926367 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:31.926371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:31.926374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:31.926404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:31.927593 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.928074 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.928077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.928079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.928158 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:31.928423 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:31.928428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:31.928430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:31.928649 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:31.928786 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.929014 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.929125 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:31.931862 19580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:31.934829 19580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:31.948909 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.943636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.98.2:39683)
0927 16:37:31.943969 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.943983 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.944156 (+ 173us) server_negotiation.cc:407] Connection header received
0927 16:37:31.944242 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.944246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.944406 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.944571 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:31.944885 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.946025 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.947548 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948208 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.948484 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.948643 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.948740 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260927 16:37:31.948951 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.942817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37687 (local address 127.0.0.1:39484)
0927 16:37:31.943148 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.943201 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.943314 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.944565 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.944568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.944606 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:31.944735 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.944741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.945855 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.945861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.947457 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.947467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948459 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.948556 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.948582 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.948637 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.948779 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":204,"spinlock_wait_cycles":3072,"thread_start_us":68,"threads_started":1}
I20260927 16:37:31.948948 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.943227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39683 (local address 127.0.0.1:51536)
0927 16:37:31.944000 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.944012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.944095 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.944559 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.944565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.944598 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:31.944743 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.944752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.946159 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.946163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.947397 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.947428 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948331 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.948444 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.948480 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.948611 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.948771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":39,"spinlock_wait_cycles":2432,"thread_start_us":90,"threads_started":1}
I20260927 16:37:31.949465 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.943479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.0.98.3:37687)
0927 16:37:31.944173 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.944187 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.944209 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:37:31.944282 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.944286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.944427 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.944588 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:31.944857 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.945733 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.947588 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948319 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948999 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.949166 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.949213 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.949299 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":170,"threads_started":1}
I20260927 16:37:31.949625 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.943013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43253 (local address 127.0.0.1:58176)
0927 16:37:31.943925 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:31.943939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:31.944029 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:31.945318 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:31.945321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:31.945338 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:31.945419 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.945425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.946724 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.946728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.947707 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:31.947717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948898 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:31.949329 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.949352 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:37:31.949425 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:37:31.949496 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":77,"thread_start_us":116,"threads_started":1}
I20260927 16:37:31.950073 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:31.943538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.0.98.1:43253)
0927 16:37:31.944886 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:31.944901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:31.944923 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:37:31.944993 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:31.944998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:31.945170 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:31.945319 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:31.945537 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.946564 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.947833 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:31.948587 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:31.948707 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:31.948841 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:31.949791 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 16:37:31.949894 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":97,"threads_started":1}
I20260927 16:37:31.952190 19839 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e00f61bd64fc78e850e82994310cb (DEFAULT_TABLE table=test-table [id=21873555f75543388c434b8ce31260ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:31.952399 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e00f61bd64fc78e850e82994310cb (DEFAULT_TABLE table=test-table [id=21873555f75543388c434b8ce31260ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:31.953312 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e00f61bd64fc78e850e82994310cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:31.953505 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e00f61bd64fc78e850e82994310cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:31.960738 20056 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Bootstrap starting.
I20260927 16:37:31.962486 20057 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Bootstrap starting.
I20260927 16:37:31.963204 20056 tablet_bootstrap.cc:654] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:31.963057 19705 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e00f61bd64fc78e850e82994310cb (DEFAULT_TABLE table=test-table [id=21873555f75543388c434b8ce31260ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:31.964160 20057 tablet_bootstrap.cc:654] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:31.964259 19705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e00f61bd64fc78e850e82994310cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:31.964751 20056 log.cc:824] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:31.965057 20057 log.cc:824] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:31.967238 20056 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: No bootstrap required, opened a new log
I20260927 16:37:31.967443 20056 ts_tablet_manager.cc:1397] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:37:31.971276 20057 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: No bootstrap required, opened a new log
I20260927 16:37:31.971503 20057 ts_tablet_manager.cc:1397] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 16:37:31.971486 20056 raft_consensus.cc:348] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.971848 20056 raft_consensus.cc:374] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:31.971912 20056 raft_consensus.cc:729] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02055c6bfba4309a758d1aaba032728, State: Initialized, Role: FOLLOWER
I20260927 16:37:31.972105 20060 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Bootstrap starting.
I20260927 16:37:31.972402 20056 consensus_queue.cc:260] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.973410 20056 ts_tablet_manager.cc:1428] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:37:31.973632 19905 heartbeater.cc:503] Master 127.0.98.62:35499 was elected leader, sending a full tablet report...
I20260927 16:37:31.974874 20060 tablet_bootstrap.cc:654] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:31.975387 20057 raft_consensus.cc:348] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.975666 20057 raft_consensus.cc:374] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:31.975723 20057 raft_consensus.cc:729] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c22e236e07145d3a25efed6d9941675, State: Initialized, Role: FOLLOWER
I20260927 16:37:31.976140 20060 log.cc:824] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:31.976155 20057 consensus_queue.cc:260] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.977119 20057 ts_tablet_manager.cc:1428] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 16:37:31.977491 20039 heartbeater.cc:503] Master 127.0.98.62:35499 was elected leader, sending a full tablet report...
I20260927 16:37:31.978253 20060 tablet_bootstrap.cc:492] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: No bootstrap required, opened a new log
I20260927 16:37:31.978443 20060 ts_tablet_manager.cc:1397] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:37:31.981103 20060 raft_consensus.cc:348] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.981323 20060 raft_consensus.cc:374] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:31.981377 20060 raft_consensus.cc:729] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6a420d33e545d8b407c92adca83131, State: Initialized, Role: FOLLOWER
I20260927 16:37:31.981788 20060 consensus_queue.cc:260] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:31.982430 19771 heartbeater.cc:503] Master 127.0.98.62:35499 was elected leader, sending a full tablet report...
I20260927 16:37:31.982610 20060 ts_tablet_manager.cc:1428] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 16:37:32.143709 20040 tablet.cc:2369] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:37:32.178221 19906 tablet.cc:2369] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:37:32.214486 19772 tablet.cc:2369] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:37:32.249974 20062 raft_consensus.cc:482] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:32.250239 20062 raft_consensus.cc:504] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:32.251572 20062 leader_election.cc:288] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd6a420d33e545d8b407c92adca83131 (127.0.98.1:43253), c02055c6bfba4309a758d1aaba032728 (127.0.98.2:39683)
I20260927 16:37:32.255689 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.251977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46121 (local address 127.0.98.1:43253)
0927 16:37:32.252129 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:32.252134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:32.252152 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:37:32.252216 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:32.252225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:32.252266 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:32.252334 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:32.252747 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.253374 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.254739 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.255133 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.255254 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.255317 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:32.255503 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 16:37:32.255568 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:37:32.255695 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.251776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43253 (local address 127.0.98.3:46121)
0927 16:37:32.251918 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:32.251933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:32.252099 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:32.252398 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:32.252404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:32.252430 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:32.252614 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.252629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.253499 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.253504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.254607 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.254617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.255263 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.255266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.255372 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.255398 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:37:32.255486 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:37:32.255559 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":33,"spinlock_wait_cycles":3200}
I20260927 16:37:32.255844 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.251958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49017 (local address 127.0.98.2:39683)
0927 16:37:32.252180 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:32.252185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:32.252454 (+ 269us) server_negotiation.cc:407] Connection header received
0927 16:37:32.252581 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:32.252585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:32.252675 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:32.252763 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:32.253061 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.253565 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.254857 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.255276 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.255396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.255458 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:32.255637 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:37:32.255703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260927 16:37:32.256251 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.251856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39683 (local address 127.0.98.3:49017)
0927 16:37:32.252304 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:32.252357 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:32.252465 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:32.252798 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:32.252801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:32.252820 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:32.252953 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.252959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.253693 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.253697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.254733 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.254743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.255396 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.255400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.255503 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.255532 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:37:32.255618 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:37:32.255686 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":17,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}
I20260927 16:37:32.256453 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e00f61bd64fc78e850e82994310cb" candidate_uuid: "0c22e236e07145d3a25efed6d9941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6a420d33e545d8b407c92adca83131" is_pre_election: true
I20260927 16:37:32.256784 19725 raft_consensus.cc:2454] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c22e236e07145d3a25efed6d9941675 in term 0.
I20260927 16:37:32.256825 19859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e00f61bd64fc78e850e82994310cb" candidate_uuid: "0c22e236e07145d3a25efed6d9941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c02055c6bfba4309a758d1aaba032728" is_pre_election: true
I20260927 16:37:32.257098 19859 raft_consensus.cc:2454] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c22e236e07145d3a25efed6d9941675 in term 0.
I20260927 16:37:32.257290 19926 leader_election.cc:302] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c22e236e07145d3a25efed6d9941675, dd6a420d33e545d8b407c92adca83131; no voters:
I20260927 16:37:32.257584 20062 raft_consensus.cc:2781] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:32.257674 20062 raft_consensus.cc:482] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:32.257725 20062 raft_consensus.cc:3037] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:32.258785 20062 raft_consensus.cc:504] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:32.259142 20062 leader_election.cc:288] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [CANDIDATE]: Term 1 election: Requested vote from peers dd6a420d33e545d8b407c92adca83131 (127.0.98.1:43253), c02055c6bfba4309a758d1aaba032728 (127.0.98.2:39683)
I20260927 16:37:32.259474 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e00f61bd64fc78e850e82994310cb" candidate_uuid: "0c22e236e07145d3a25efed6d9941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6a420d33e545d8b407c92adca83131"
I20260927 16:37:32.259565 19859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e00f61bd64fc78e850e82994310cb" candidate_uuid: "0c22e236e07145d3a25efed6d9941675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c02055c6bfba4309a758d1aaba032728"
I20260927 16:37:32.259603 19725 raft_consensus.cc:3037] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:32.259685 19859 raft_consensus.cc:3037] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:32.260907 19725 raft_consensus.cc:2454] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c22e236e07145d3a25efed6d9941675 in term 1.
I20260927 16:37:32.260907 19859 raft_consensus.cc:2454] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c22e236e07145d3a25efed6d9941675 in term 1.
I20260927 16:37:32.261205 19926 leader_election.cc:302] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c22e236e07145d3a25efed6d9941675, dd6a420d33e545d8b407c92adca83131; no voters:
I20260927 16:37:32.261412 20062 raft_consensus.cc:2781] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:32.261654 20062 raft_consensus.cc:686] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 1 LEADER]: Becoming Leader. State: Replica: 0c22e236e07145d3a25efed6d9941675, State: Running, Role: LEADER
I20260927 16:37:32.261976 20062 consensus_queue.cc:237] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } }
I20260927 16:37:32.264953 19580 catalog_manager.cc:5495] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c22e236e07145d3a25efed6d9941675 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0c22e236e07145d3a25efed6d9941675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c22e236e07145d3a25efed6d9941675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 } health_report { overall_health: UNKNOWN } } }
I20260927 16:37:32.312408 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.309204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.98.3:37687)
0927 16:37:32.309342 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:32.309346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:32.309358 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:37:32.309433 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:32.309435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:32.309490 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:32.309577 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:32.309798 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.310326 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.311121 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.311331 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.311402 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.311666 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:32.312088 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:32.312192 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:32.312251 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:37:32.312300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:37:32.318579 19859 raft_consensus.cc:1261] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 1 FOLLOWER]: Refusing update from remote peer 0c22e236e07145d3a25efed6d9941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:32.318614 19725 raft_consensus.cc:1261] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 1 FOLLOWER]: Refusing update from remote peer 0c22e236e07145d3a25efed6d9941675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:32.319334 20062 consensus_queue.cc:1035] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6a420d33e545d8b407c92adca83131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:32.319533 20066 consensus_queue.cc:1035] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c02055c6bfba4309a758d1aaba032728" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:32.325904 20069 mvcc.cc:205] Tried to move back new op lower bound from 7333998806288306176 to 7333998806067519488. Current Snapshot: MvccSnapshot[applied={T|T < 7333998806288306176}]
I20260927 16:37:32.327055 20071 mvcc.cc:205] Tried to move back new op lower bound from 7333998806288306176 to 7333998806067519488. Current Snapshot: MvccSnapshot[applied={T|T < 7333998806288306176}]
I20260927 16:37:32.336009 19580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:32.336212 19580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:32.338629 19580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8539fb61f764e2ca2336461c167cb62: Sending DeleteTablet for 3 replicas of tablet 6f8e00f61bd64fc78e850e82994310cb
I20260927 16:37:32.339507 19705 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8e00f61bd64fc78e850e82994310cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260927 16:37:32.339565 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8e00f61bd64fc78e850e82994310cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51536
I20260927 16:37:32.340013 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8e00f61bd64fc78e850e82994310cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39484
I20260927 16:37:32.340058 20079 tablet_replica.cc:330] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131: stopping tablet replica
I20260927 16:37:32.340513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19642
I20260927 16:37:32.340512 20079 raft_consensus.cc:2229] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:32.340837 20079 raft_consensus.cc:2258] T 6f8e00f61bd64fc78e850e82994310cb P dd6a420d33e545d8b407c92adca83131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:32.341192 20080 tablet_replica.cc:330] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: stopping tablet replica
I20260927 16:37:32.341501 20080 raft_consensus.cc:2229] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:32.341848 20080 raft_consensus.cc:2258] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:32.344736 20080 ts_tablet_manager.cc:1905] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:32.346387 20081 tablet_replica.cc:330] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: stopping tablet replica
I20260927 16:37:32.346733 20081 raft_consensus.cc:2229] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:37:32.347188 20081 raft_consensus.cc:2258] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:32.348325 20080 ts_tablet_manager.cc:1918] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:32.348438 20080 log.cc:1196] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-1/wal/wals/6f8e00f61bd64fc78e850e82994310cb
I20260927 16:37:32.348769 20080 ts_tablet_manager.cc:1939] T 6f8e00f61bd64fc78e850e82994310cb P c02055c6bfba4309a758d1aaba032728: Deleting consensus metadata
I20260927 16:37:32.348894 20081 ts_tablet_manager.cc:1905] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:37:32.349596 19564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43253 (error 108)
I20260927 16:37:32.349750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19776
I20260927 16:37:32.351274 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.350928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43253 (local address 127.0.0.1:58180)
0927 16:37:32.351110 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:32.351185 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 16:37:32.351449 19564 catalog_manager.cc:4584] TS dd6a420d33e545d8b407c92adca83131 (127.0.98.1:43253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f8e00f61bd64fc78e850e82994310cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43253: connect: Connection refused (error 111)
I20260927 16:37:32.353318 20081 ts_tablet_manager.cc:1918] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:32.353443 20081 log.cc:1196] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/6f8e00f61bd64fc78e850e82994310cb
I20260927 16:37:32.353853 20081 ts_tablet_manager.cc:1939] T 6f8e00f61bd64fc78e850e82994310cb P 0c22e236e07145d3a25efed6d9941675: Deleting consensus metadata
I20260927 16:37:32.355114 19565 catalog_manager.cc:4841] TS 0c22e236e07145d3a25efed6d9941675 (127.0.98.3:37687): tablet 6f8e00f61bd64fc78e850e82994310cb (table test-table [id=21873555f75543388c434b8ce31260ca]) successfully deleted
I20260927 16:37:32.356459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19910
I20260927 16:37:32.358083 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.357779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39683 (local address 127.0.0.1:51538)
0927 16:37:32.357912 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:32.357977 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:37:32.362520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 19548
2026-09-27T16:37:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2531 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T16:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:32Z Disabled control of system clock
I20260927 16:37:32.392763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32841
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:32.504328 20086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:32.504594 20086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:32.504628 20086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:32.508793 20086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:32.508850 20086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:32.508865 20086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:32.508882 20086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:32.508898 20086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:32.513572 20086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:32.514772 20086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:32.516157 20086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:32.524956 20086 server_base.cc:1034] running on GCE node
W20260927 16:37:32.524907 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.524916 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.524907 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:32.525704 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:32.526858 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:32.528014 20086 hybrid_clock.cc:648] HybridClock initialized: now 1790527052527977 us; error 50 us; skew 500 ppm
I20260927 16:37:32.529933 20086 webserver.cc:460] Webserver started at http://127.0.98.62:44763/ using document root <none> and password file <none>
I20260927 16:37:32.530565 20086 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:32.530627 20086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:32.530856 20086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:32.532876 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "1130a6f674c44fd9ae676df517a9ddfc"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "cfcfb29d6148c4bda0b6d3c7d35f47cc"
server_key_iv: "32dee538792f1d4a7b7fbe7a9aeb2bef"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.533442 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "1130a6f674c44fd9ae676df517a9ddfc"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "cfcfb29d6148c4bda0b6d3c7d35f47cc"
server_key_iv: "32dee538792f1d4a7b7fbe7a9aeb2bef"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.537065 20086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:37:32.539479 20100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:32.540630 20086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:32.540772 20086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "1130a6f674c44fd9ae676df517a9ddfc"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "cfcfb29d6148c4bda0b6d3c7d35f47cc"
server_key_iv: "32dee538792f1d4a7b7fbe7a9aeb2bef"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.540877 20086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:32.564240 20086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:32.565016 20086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:32.565225 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:32.572484 20086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32841
I20260927 16:37:32.572489 20152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32841 every 8 connection(s)
I20260927 16:37:32.573455 20086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:32.576506 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:32.578790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20086
I20260927 16:37:32.578917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:32.579214 392 external_mini_cluster.cc:1428] Setting key e5e598b74b62ee978a9cf9edf9756de6
I20260927 16:37:32.582345 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: Bootstrap starting.
I20260927 16:37:32.584779 20153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:32.585551 20153 log.cc:824] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:32.587751 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: No bootstrap required, opened a new log
I20260927 16:37:32.588462 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.580625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.98.62:32841)
0927 16:37:32.581233 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:32.581249 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:32.581284 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:32.581886 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:32.581903 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:32.582360 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:32.582670 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:32.583015 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.583945 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.585151 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.585687 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.585805 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.586755 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:32.586760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:32.586772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:32.587138 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:32.587204 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:32.587223 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:32.587296 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:32.587456 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:37:32.587737 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":129,"threads_started":1}
I20260927 16:37:32.591223 20153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } }
I20260927 16:37:32.591426 20153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:32.591502 20153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1130a6f674c44fd9ae676df517a9ddfc, State: Initialized, Role: FOLLOWER
I20260927 16:37:32.591908 20153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } }
I20260927 16:37:32.592047 20153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:32.592109 20153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:32.592212 20153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:32.593115 20153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } }
I20260927 16:37:32.593463 20153 leader_election.cc:302] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1130a6f674c44fd9ae676df517a9ddfc; no voters:
I20260927 16:37:32.593824 20153 leader_election.cc:288] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:32.593910 20158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:32.594173 20158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [term 1 LEADER]: Becoming Leader. State: Replica: 1130a6f674c44fd9ae676df517a9ddfc, State: Running, Role: LEADER
I20260927 16:37:32.594530 20158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } }
I20260927 16:37:32.595255 20153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:32.596185 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1130a6f674c44fd9ae676df517a9ddfc. Latest consensus state: current_term: 1 leader_uuid: "1130a6f674c44fd9ae676df517a9ddfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } } }
I20260927 16:37:32.596305 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:32.596233 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1130a6f674c44fd9ae676df517a9ddfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1130a6f674c44fd9ae676df517a9ddfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32841 } } }
I20260927 16:37:32.596464 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:32.596727 20167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:32.599586 20167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:32.604755 20167 catalog_manager.cc:1347] Generated new cluster ID: 6312e7e68c084437a6beda99f6f53580
I20260927 16:37:32.604818 20167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:32.629405 20167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:32.630790 20167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:32.643779 20167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: Generated new TSK 0
I20260927 16:37:32.644742 20167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:32.656013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--builtin_ntp_servers=127.0.98.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:37:32.770007 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:32.770359 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:32.770403 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:32.774551 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:32.774611 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:32.774672 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:37:32.779502 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:32.780592 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:32.782042 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:32.792237 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.792271 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.792237 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:32.792299 20177 server_base.cc:1034] running on GCE node
I20260927 16:37:32.792906 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:32.793633 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:32.794823 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790527052794806 us; error 31 us; skew 500 ppm
I20260927 16:37:32.796988 20177 webserver.cc:460] Webserver started at http://127.0.98.1:45233/ using document root <none> and password file <none>
I20260927 16:37:32.797675 20177 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:32.797781 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:32.798046 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:32.800221 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fec645e13d44a0e9db4bb5cef9d9190"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "ee3aab3d62ed6eea2548d3a9c28c0d42"
server_key_iv: "1f08f13f43a9933fc5061ff5037705e2"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.800778 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fec645e13d44a0e9db4bb5cef9d9190"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "ee3aab3d62ed6eea2548d3a9c28c0d42"
server_key_iv: "1f08f13f43a9933fc5061ff5037705e2"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.804414 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:37:32.806735 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:32.807927 20177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:32.808064 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal
uuid: "4fec645e13d44a0e9db4bb5cef9d9190"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "ee3aab3d62ed6eea2548d3a9c28c0d42"
server_key_iv: "1f08f13f43a9933fc5061ff5037705e2"
server_key_version: "encryptionkey@0"
I20260927 16:37:32.808175 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:32.839311 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:32.840067 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:32.840272 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:32.840873 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:32.841826 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:32.841897 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:32.841969 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:32.842020 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:32.850714 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37945
I20260927 16:37:32.850740 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37945 every 8 connection(s)
I20260927 16:37:32.851570 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:37:32.852563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20177
I20260927 16:37:32.852689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:37:32.853000 392 external_mini_cluster.cc:1428] Setting key c410811748c744c00f62f983e8a62768
I20260927 16:37:32.855043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--builtin_ntp_servers=127.0.98.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:32.862058 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.853577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40993 (local address 127.0.98.62:32841)
0927 16:37:32.853707 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:32.853711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:32.854943 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 16:37:32.856387 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:32.856391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:32.856444 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:32.856524 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:32.857555 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.858074 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.859988 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.860211 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.860287 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.861599 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:32.861602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:32.861604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:32.861615 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:32.861635 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:32.861643 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:32.861700 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:32.861898 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 16:37:32.861948 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:37:32.863356 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:32.854020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32841 (local address 127.0.98.1:40993)
0927 16:37:32.854804 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:32.854844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:32.856178 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:32.856691 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:32.856698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:32.856782 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:32.857400 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.857421 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.858235 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.858242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.859849 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:32.859864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:32.860368 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:32.860374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:32.860613 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:32.861287 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:32.861289 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:32.861351 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:32.861440 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:32.861733 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:32.861744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:32.861754 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 16:37:32.861898 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 16:37:32.862258 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":221,"threads_started":1}
I20260927 16:37:32.864606 20305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32841
I20260927 16:37:32.864868 20305 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:32.865523 20305 heartbeater.cc:511] Master 127.0.98.62:32841 requested a full tablet report, sending...
I20260927 16:37:32.867205 20117 ts_manager.cc:194] Registered new tserver with Master: 4fec645e13d44a0e9db4bb5cef9d9190 (127.0.98.1:37945)
I20260927 16:37:32.868705 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40993
W20260927 16:37:32.972077 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:32.972338 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:32.972376 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:32.976339 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:32.976390 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:32.976444 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:37:32.981165 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:32.982153 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:32.983525 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:32.992916 20308 server_base.cc:1034] running on GCE node
W20260927 16:37:32.992882 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.992887 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:32.992882 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:32.993578 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:32.994158 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:32.995405 20308 hybrid_clock.cc:648] HybridClock initialized: now 1790527052995387 us; error 30 us; skew 500 ppm
I20260927 16:37:32.997840 20308 webserver.cc:460] Webserver started at http://127.0.98.2:38647/ using document root <none> and password file <none>
I20260927 16:37:32.998588 20308 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:32.998687 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:32.998939 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:33.001245 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data/instance:
uuid: "d664b009147043c09325d89a9ede703a"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "2e7fc1fbb1a1a92143e053f22b08cace"
server_key_iv: "70ca1eeee8545bfb5386356b05ee4385"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.001860 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d664b009147043c09325d89a9ede703a"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "2e7fc1fbb1a1a92143e053f22b08cace"
server_key_iv: "70ca1eeee8545bfb5386356b05ee4385"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.005761 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 16:37:33.008327 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.009604 20308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:37:33.009742 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal
uuid: "d664b009147043c09325d89a9ede703a"
format_stamp: "Formatted at 2026-09-27 16:37:32 on dist-test-slave-t27c"
server_key: "2e7fc1fbb1a1a92143e053f22b08cace"
server_key_iv: "70ca1eeee8545bfb5386356b05ee4385"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.009852 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:33.056483 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:33.057236 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:33.057436 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:33.058135 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:33.059398 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:33.059468 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.059533 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:33.059564 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.068423 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38835
I20260927 16:37:33.068439 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38835 every 8 connection(s)
I20260927 16:37:33.069344 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:37:33.071707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20308
I20260927 16:37:33.071852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:37:33.072180 392 external_mini_cluster.cc:1428] Setting key 0455ebd19b8b830b69ca79d80122e0e4
I20260927 16:37:33.074438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--builtin_ntp_servers=127.0.98.20:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:37:33.080505 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.071243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50235 (local address 127.0.98.62:32841)
0927 16:37:33.071386 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.071390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.072571 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 16:37:33.074350 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.074353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.074406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.074495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:33.075870 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.076825 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.078087 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.078541 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.078674 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.079863 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:33.079870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:33.079876 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:33.079897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:33.079925 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:33.079937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:33.080037 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.080195 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.080276 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:37:33.081465 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.071658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32841 (local address 127.0.98.2:50235)
0927 16:37:33.072382 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.072436 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.074046 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.074741 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.074753 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.074887 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:33.075674 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.075697 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.076974 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.076977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.077944 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.077954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.078698 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.078701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.078869 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.079576 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:33.079578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:33.079622 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:33.079692 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:33.080032 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:33.080039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:33.080043 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.080159 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.080393 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260927 16:37:33.082806 20436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32841
I20260927 16:37:33.083097 20436 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:33.083811 20436 heartbeater.cc:511] Master 127.0.98.62:32841 requested a full tablet report, sending...
I20260927 16:37:33.085068 20117 ts_manager.cc:194] Registered new tserver with Master: d664b009147043c09325d89a9ede703a (127.0.98.2:38835)
I20260927 16:37:33.085667 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50235
W20260927 16:37:33.187244 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:33.187505 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:33.187599 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:33.191648 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:33.191705 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:33.191794 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:37:33.196951 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:33.198308 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:33.199765 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:33.209309 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:33.209309 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:33.209329 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:33.209384 20439 server_base.cc:1034] running on GCE node
I20260927 16:37:33.209918 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:33.210667 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:33.211848 20439 hybrid_clock.cc:648] HybridClock initialized: now 1790527053211833 us; error 27 us; skew 500 ppm
I20260927 16:37:33.213876 20439 webserver.cc:460] Webserver started at http://127.0.98.3:37387/ using document root <none> and password file <none>
I20260927 16:37:33.214534 20439 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:33.214623 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:33.214828 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:33.216823 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data/instance:
uuid: "94b5014dd2e74c969c4a48824b14c617"
format_stamp: "Formatted at 2026-09-27 16:37:33 on dist-test-slave-t27c"
server_key: "b65eafc68d2d52e9534546b85155b18f"
server_key_iv: "c5edd0c119787b6e55c35f4df0f19c9b"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.217386 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94b5014dd2e74c969c4a48824b14c617"
format_stamp: "Formatted at 2026-09-27 16:37:33 on dist-test-slave-t27c"
server_key: "b65eafc68d2d52e9534546b85155b18f"
server_key_iv: "c5edd0c119787b6e55c35f4df0f19c9b"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.221076 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:37:33.223604 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.224915 20439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:33.225054 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal
uuid: "94b5014dd2e74c969c4a48824b14c617"
format_stamp: "Formatted at 2026-09-27 16:37:33 on dist-test-slave-t27c"
server_key: "b65eafc68d2d52e9534546b85155b18f"
server_key_iv: "c5edd0c119787b6e55c35f4df0f19c9b"
server_key_version: "encryptionkey@0"
I20260927 16:37:33.225162 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:33.253829 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:33.254657 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:33.254866 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:33.255520 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:37:33.256500 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:37:33.256578 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.256647 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:37:33.256700 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:33.265372 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38063
I20260927 16:37:33.265403 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38063 every 8 connection(s)
I20260927 16:37:33.266333 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:37:33.271490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20439
I20260927 16:37:33.271617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:37:33.271919 392 external_mini_cluster.cc:1428] Setting key 9c7485eca70778c3796f6c927b7f9ba5
I20260927 16:37:33.276774 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.268176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47935 (local address 127.0.98.62:32841)
0927 16:37:33.268321 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.268325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.269385 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 16:37:33.270947 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.270951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.271041 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.271168 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:33.272347 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.273125 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.274679 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.274933 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.275077 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.276259 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:33.276264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:33.276272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:33.276284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:33.276302 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:33.276310 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:33.276393 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.276559 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.276624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:37:33.277731 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.268582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32841 (local address 127.0.98.3:47935)
0927 16:37:33.269238 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.269279 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.270662 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.271443 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.271456 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.271584 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:37:33.272201 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.272216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.273304 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.273309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.274488 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.274530 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.275120 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.275125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.275317 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.275972 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:37:33.275973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:37:33.276020 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:37:33.276089 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:37:33.276394 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:37:33.276405 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:37:33.276414 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.276549 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.276879 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260927 16:37:33.278939 20567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32841
I20260927 16:37:33.279208 20567 heartbeater.cc:464] Registering TS with master...
I20260927 16:37:33.279742 20567 heartbeater.cc:511] Master 127.0.98.62:32841 requested a full tablet report, sending...
I20260927 16:37:33.280745 20117 ts_manager.cc:194] Registered new tserver with Master: 94b5014dd2e74c969c4a48824b14c617 (127.0.98.3:38063)
I20260927 16:37:33.281286 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47935
I20260927 16:37:33.285957 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:37:33.291182 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.287864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.62:32841)
0927 16:37:33.287974 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.287977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.288172 (+ 195us) server_negotiation.cc:407] Connection header received
0927 16:37:33.288259 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.288262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.288299 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.288357 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:33.288561 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.289042 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.289815 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.290012 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.290084 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.290889 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:33.290892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:33.290894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:33.290903 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:33.290918 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:33.290925 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:33.290979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.291027 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.291070 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:37:33.293884 20117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:37:33.297201 20117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:37:33.311815 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.305739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.0.98.2:38835)
0927 16:37:33.306126 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.306138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.306863 (+ 725us) server_negotiation.cc:407] Connection header received
0927 16:37:33.307079 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.307087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.307308 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.307469 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:33.307679 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308827 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.310524 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311123 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311247 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311435 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.311540 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.311636 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
I20260927 16:37:33.311868 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.305544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38835 (local address 127.0.0.1:33222)
0927 16:37:33.306776 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.306789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.306924 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.307451 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.307454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.307470 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:33.307567 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.307572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308984 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.308987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.310386 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.310396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311247 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.311356 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311392 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.311511 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.311634 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":163,"thread_start_us":97,"threads_started":1}
I20260927 16:37:33.312084 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.306195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38063 (local address 127.0.0.1:48540)
0927 16:37:33.307133 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.307159 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.307237 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.307727 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.307729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.307745 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:33.307828 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.307834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308797 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.308803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.310130 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.310139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311501 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.311785 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311803 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.311864 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.311953 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":105,"thread_start_us":89,"threads_started":1}
I20260927 16:37:33.312259 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.306219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.0.98.1:37945)
0927 16:37:33.306609 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.306621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.306648 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:37:33.306819 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.306828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.307056 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.307228 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:33.307548 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308824 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.310486 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311222 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311308 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311446 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.311985 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.312080 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260927 16:37:33.312587 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.305444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:43268)
0927 16:37:33.306424 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.306475 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.306601 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.307223 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.307228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.307254 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:33.307411 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.307422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308984 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.308987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.310353 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.310365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311501 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.311712 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311785 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.311861 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.312460 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":628,"thread_start_us":91,"threads_started":1}
I20260927 16:37:33.313176 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.306626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.3:38063)
0927 16:37:33.307058 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.307069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.307324 (+ 255us) server_negotiation.cc:407] Connection header received
0927 16:37:33.307426 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.307433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.307618 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.307755 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:33.307938 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.308674 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.310303 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.311114 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.311229 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.311435 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.312870 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.312969 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":107,"threads_started":1}
I20260927 16:37:33.315351 20370 tablet_service.cc:1467] Processing CreateTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d (DEFAULT_TABLE table=test-table [id=2db77f0f518d457a9f1e2d79018064bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:33.315584 20501 tablet_service.cc:1467] Processing CreateTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d (DEFAULT_TABLE table=test-table [id=2db77f0f518d457a9f1e2d79018064bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:33.316399 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb92b2214eaa4f85b655a9d9f08f756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:33.316471 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb92b2214eaa4f85b655a9d9f08f756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:33.320703 20239 tablet_service.cc:1467] Processing CreateTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d (DEFAULT_TABLE table=test-table [id=2db77f0f518d457a9f1e2d79018064bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:37:33.321887 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb92b2214eaa4f85b655a9d9f08f756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:33.323110 20584 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Bootstrap starting.
I20260927 16:37:33.324540 20585 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Bootstrap starting.
I20260927 16:37:33.325793 20584 tablet_bootstrap.cc:654] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:33.326639 20585 tablet_bootstrap.cc:654] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:33.326715 20584 log.cc:824] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:33.327471 20585 log.cc:824] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:33.329336 20584 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: No bootstrap required, opened a new log
I20260927 16:37:33.329336 20585 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: No bootstrap required, opened a new log
I20260927 16:37:33.329532 20584 ts_tablet_manager.cc:1397] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:37:33.329532 20585 ts_tablet_manager.cc:1397] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:37:33.330499 20588 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Bootstrap starting.
I20260927 16:37:33.332250 20588 tablet_bootstrap.cc:654] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:33.332911 20588 log.cc:824] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:33.333949 20584 raft_consensus.cc:348] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.333949 20585 raft_consensus.cc:348] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.334254 20584 raft_consensus.cc:374] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:33.334312 20584 raft_consensus.cc:729] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d664b009147043c09325d89a9ede703a, State: Initialized, Role: FOLLOWER
I20260927 16:37:33.334264 20585 raft_consensus.cc:374] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:33.334678 20585 raft_consensus.cc:729] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b5014dd2e74c969c4a48824b14c617, State: Initialized, Role: FOLLOWER
I20260927 16:37:33.334806 20588 tablet_bootstrap.cc:492] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: No bootstrap required, opened a new log
I20260927 16:37:33.334798 20584 consensus_queue.cc:260] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.334987 20588 ts_tablet_manager.cc:1397] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:37:33.335563 20436 heartbeater.cc:503] Master 127.0.98.62:32841 was elected leader, sending a full tablet report...
I20260927 16:37:33.335835 20584 ts_tablet_manager.cc:1428] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:37:33.335429 20585 consensus_queue.cc:260] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.336706 20567 heartbeater.cc:503] Master 127.0.98.62:32841 was elected leader, sending a full tablet report...
I20260927 16:37:33.336977 20585 ts_tablet_manager.cc:1428] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 16:37:33.339465 20588 raft_consensus.cc:348] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.339728 20588 raft_consensus.cc:374] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:33.339787 20588 raft_consensus.cc:729] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fec645e13d44a0e9db4bb5cef9d9190, State: Initialized, Role: FOLLOWER
I20260927 16:37:33.340198 20588 consensus_queue.cc:260] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.340873 20305 heartbeater.cc:503] Master 127.0.98.62:32841 was elected leader, sending a full tablet report...
I20260927 16:37:33.341053 20588 ts_tablet_manager.cc:1428] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 16:37:33.352691 20306 tablet.cc:2369] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:37:33.470921 20591 raft_consensus.cc:482] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:37:33.471151 20591 raft_consensus.cc:504] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.472376 20591 leader_election.cc:288] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fec645e13d44a0e9db4bb5cef9d9190 (127.0.98.1:37945), d664b009147043c09325d89a9ede703a (127.0.98.2:38835)
I20260927 16:37:33.477403 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.472799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51849 (local address 127.0.98.2:38835)
0927 16:37:33.472937 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.472942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.472965 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:37:33.473057 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.473060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.473117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.473206 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:33.473530 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.474369 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476051 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.476680 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476787 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.476856 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.477184 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.477261 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:37:33.477403 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.472788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60651 (local address 127.0.98.1:37945)
0927 16:37:33.472951 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.472957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.473369 (+ 412us) server_negotiation.cc:407] Connection header received
0927 16:37:33.473460 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.473482 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.473538 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.473617 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:37:33.473849 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.474589 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476051 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.476672 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.476856 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.477186 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.477261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 16:37:33.477425 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.472684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38835 (local address 127.0.98.3:51849)
0927 16:37:33.472818 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.472836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.472949 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.473243 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.473246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.473263 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:33.473409 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.473419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.474482 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.474485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.475904 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.475917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.476805 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.477012 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.477064 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.477147 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.477251 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":194}
I20260927 16:37:33.477433 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.472679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.98.3:60651)
0927 16:37:33.473262 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.473275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:37:33.473367 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:37:33.473632 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:37:33.473634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:37:33.473649 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:37:33.473740 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.473744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.474687 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.474690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.475916 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:37:33.475922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.476805 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.476808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:37:33.477007 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.477073 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 16:37:33.477163 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:37:33.477251 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":258,"thread_start_us":157,"threads_started":1}
I20260927 16:37:33.478307 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb92b2214eaa4f85b655a9d9f08f756d" candidate_uuid: "94b5014dd2e74c969c4a48824b14c617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d664b009147043c09325d89a9ede703a" is_pre_election: true
I20260927 16:37:33.478343 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb92b2214eaa4f85b655a9d9f08f756d" candidate_uuid: "94b5014dd2e74c969c4a48824b14c617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" is_pre_election: true
I20260927 16:37:33.478632 20259 raft_consensus.cc:2454] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b5014dd2e74c969c4a48824b14c617 in term 0.
I20260927 16:37:33.478638 20390 raft_consensus.cc:2454] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b5014dd2e74c969c4a48824b14c617 in term 0.
I20260927 16:37:33.479209 20457 leader_election.cc:302] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4fec645e13d44a0e9db4bb5cef9d9190, 94b5014dd2e74c969c4a48824b14c617, d664b009147043c09325d89a9ede703a; no voters:
I20260927 16:37:33.479485 20591 raft_consensus.cc:2781] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:37:33.479563 20591 raft_consensus.cc:482] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:37:33.479614 20591 raft_consensus.cc:3037] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:33.480687 20591 raft_consensus.cc:504] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.481076 20591 leader_election.cc:288] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [CANDIDATE]: Term 1 election: Requested vote from peers 4fec645e13d44a0e9db4bb5cef9d9190 (127.0.98.1:37945), d664b009147043c09325d89a9ede703a (127.0.98.2:38835)
I20260927 16:37:33.481401 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb92b2214eaa4f85b655a9d9f08f756d" candidate_uuid: "94b5014dd2e74c969c4a48824b14c617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fec645e13d44a0e9db4bb5cef9d9190"
I20260927 16:37:33.481444 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb92b2214eaa4f85b655a9d9f08f756d" candidate_uuid: "94b5014dd2e74c969c4a48824b14c617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d664b009147043c09325d89a9ede703a"
I20260927 16:37:33.481513 20259 raft_consensus.cc:3037] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:33.481590 20390 raft_consensus.cc:3037] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:33.482722 20259 raft_consensus.cc:2454] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94b5014dd2e74c969c4a48824b14c617 in term 1.
I20260927 16:37:33.482765 20390 raft_consensus.cc:2454] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94b5014dd2e74c969c4a48824b14c617 in term 1.
I20260927 16:37:33.483096 20457 leader_election.cc:302] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fec645e13d44a0e9db4bb5cef9d9190, 94b5014dd2e74c969c4a48824b14c617; no voters:
I20260927 16:37:33.483292 20591 raft_consensus.cc:2781] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:33.483518 20591 raft_consensus.cc:686] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 1 LEADER]: Becoming Leader. State: Replica: 94b5014dd2e74c969c4a48824b14c617, State: Running, Role: LEADER
I20260927 16:37:33.483841 20591 consensus_queue.cc:237] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } }
I20260927 16:37:33.487126 20117 catalog_manager.cc:5495] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94b5014dd2e74c969c4a48824b14c617 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94b5014dd2e74c969c4a48824b14c617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94b5014dd2e74c969c4a48824b14c617" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38063 } health_report { overall_health: HEALTHY } } }
I20260927 16:37:33.500406 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.497214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.98.3:38063)
0927 16:37:33.497315 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:33.497318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:33.497331 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:37:33.497407 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:33.497409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:33.497458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:33.497535 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:37:33.497739 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.498284 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.499101 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:33.499299 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:33.499371 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:33.499621 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:37:33.500023 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:37:33.500110 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:33.500169 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:37:33.500209 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:37:33.506150 20390 raft_consensus.cc:1261] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 1 FOLLOWER]: Refusing update from remote peer 94b5014dd2e74c969c4a48824b14c617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:33.506150 20259 raft_consensus.cc:1261] T bb92b2214eaa4f85b655a9d9f08f756d P 4fec645e13d44a0e9db4bb5cef9d9190 [term 1 FOLLOWER]: Refusing update from remote peer 94b5014dd2e74c969c4a48824b14c617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:37:33.506914 20591 consensus_queue.cc:1035] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d664b009147043c09325d89a9ede703a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:33.507103 20594 consensus_queue.cc:1035] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fec645e13d44a0e9db4bb5cef9d9190" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:37:33.516556 20598 mvcc.cc:205] Tried to move back new op lower bound from 7333998811153829888 to 7333998811072503808. Current Snapshot: MvccSnapshot[applied={T|T < 7333998811153829888}]
I20260927 16:37:33.522898 20116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:33.523056 20116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:37:33.525506 20116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1130a6f674c44fd9ae676df517a9ddfc: Sending DeleteTablet for 3 replicas of tablet bb92b2214eaa4f85b655a9d9f08f756d
I20260927 16:37:33.526340 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:33 UTC) from {username='slave'} at 127.0.0.1:33222
I20260927 16:37:33.526377 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:33 UTC) from {username='slave'} at 127.0.0.1:48540
I20260927 16:37:33.526444 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet bb92b2214eaa4f85b655a9d9f08f756d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:37:33 UTC) from {username='slave'} at 127.0.0.1:43268
I20260927 16:37:33.526837 20608 tablet_replica.cc:330] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: stopping tablet replica
I20260927 16:37:33.527156 20609 tablet_replica.cc:330] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: stopping tablet replica
I20260927 16:37:33.527421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20177
I20260927 16:37:33.527469 20609 raft_consensus.cc:2229] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:37:33.527840 20609 raft_consensus.cc:2258] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:33.529323 20609 ts_tablet_manager.cc:1905] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:37:33.532912 20609 ts_tablet_manager.cc:1918] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:37:33.533020 20609 log.cc:1196] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Deleting WAL directory at /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526802550915-392-0/minicluster-data/ts-2/wal/wals/bb92b2214eaa4f85b655a9d9f08f756d
I20260927 16:37:33.533339 20609 ts_tablet_manager.cc:1939] T bb92b2214eaa4f85b655a9d9f08f756d P 94b5014dd2e74c969c4a48824b14c617: Deleting consensus metadata
I20260927 16:37:33.534619 20104 catalog_manager.cc:4841] TS 94b5014dd2e74c969c4a48824b14c617 (127.0.98.3:38063): tablet bb92b2214eaa4f85b655a9d9f08f756d (table test-table [id=2db77f0f518d457a9f1e2d79018064bc]) successfully deleted
I20260927 16:37:33.534919 20608 raft_consensus.cc:2229] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:37:33.535267 20608 raft_consensus.cc:2258] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:37:33.536818 20608 ts_tablet_manager.cc:1905] T bb92b2214eaa4f85b655a9d9f08f756d P d664b009147043c09325d89a9ede703a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:37:33.536859 20104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37945 (error 108)
I20260927 16:37:33.537920 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.537591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:43276)
0927 16:37:33.537737 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.537816 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:37:33.538002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20308
W20260927 16:37:33.540565 20104 catalog_manager.cc:4584] TS 4fec645e13d44a0e9db4bb5cef9d9190 (127.0.98.1:37945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb92b2214eaa4f85b655a9d9f08f756d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
I20260927 16:37:33.545261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20439
I20260927 16:37:33.545466 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:33.545178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38835 (local address 127.0.0.1:33238)
0927 16:37:33.545300 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:37:33.545366 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 16:37:33.550820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20086
2026-09-27T16:37:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1186 ms)
[----------] 6 tests from EncryptionPolicyTest (10502 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:33 dist-test-slave-t27c krb5kdc[20614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:33 dist-test-slave-t27c krb5kdc[20614](info): set up 2 sockets
Sep 27 16:37:33 dist-test-slave-t27c krb5kdc[20614](info): commencing operation
krb5kdc: starting...
W20260927 16:37:34.590289 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:34 dist-test-slave-t27c krb5kdc[20614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:34.713546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:34.825076 20627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:34.825346 20627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:34.825389 20627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:34.829452 20627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:34.829516 20627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:34.829535 20627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:34.829551 20627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:34.829566 20627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:34.834919 20627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:34.836117 20627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:34.837284 20627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:34.846372 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:34.846413 20627 server_base.cc:1034] running on GCE node
W20260927 16:37:34.846364 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:34.846366 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:34.847157 20627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:34.848282 20627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:34.849439 20627 hybrid_clock.cc:648] HybridClock initialized: now 1790527054849409 us; error 41 us; skew 500 ppm
Sep 27 16:37:34 dist-test-slave-t27c krb5kdc[20614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:34.852164 20627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:34.853073 20627 webserver.cc:460] Webserver started at http://127.0.0.1:32841/ using document root <none> and password file <none>
I20260927 16:37:34.853631 20627 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:34.853744 20627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:34.853973 20627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:34.856112 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "70f868bb51f4495196ec086df2bbc33c"
format_stamp: "Formatted at 2026-09-27 16:37:34 on dist-test-slave-t27c"
server_key: "e483fe8ce746340bc41f84755ae1dd81"
server_key_iv: "5b0a24c7fc368cc2aa5c60a2a13825c0"
server_key_version: "encryptionkey@0"
I20260927 16:37:34.856679 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "70f868bb51f4495196ec086df2bbc33c"
format_stamp: "Formatted at 2026-09-27 16:37:34 on dist-test-slave-t27c"
server_key: "e483fe8ce746340bc41f84755ae1dd81"
server_key_iv: "5b0a24c7fc368cc2aa5c60a2a13825c0"
server_key_version: "encryptionkey@0"
I20260927 16:37:34.860343 20627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:37:34.862782 20642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:34.863993 20627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:34.864132 20627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "70f868bb51f4495196ec086df2bbc33c"
format_stamp: "Formatted at 2026-09-27 16:37:34 on dist-test-slave-t27c"
server_key: "e483fe8ce746340bc41f84755ae1dd81"
server_key_iv: "5b0a24c7fc368cc2aa5c60a2a13825c0"
server_key_version: "encryptionkey@0"
I20260927 16:37:34.864243 20627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:34.874965 20627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:34.878631 20627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:34.878856 20627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:34.886631 20627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44769
I20260927 16:37:34.886660 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44769 every 8 connection(s)
I20260927 16:37:34.887633 20627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:34.890039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20627
I20260927 16:37:34.890208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:34.890515 392 external_mini_cluster.cc:1428] Setting key cea9d4a6cd6c1e21ee35ae5f70cbf7ab
I20260927 16:37:34.890587 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:34.896883 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c: Bootstrap starting.
I20260927 16:37:34.899415 20695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c: Neither blocks nor log segments found. Creating new log.
Sep 27 16:37:34 dist-test-slave-t27c krb5kdc[20614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:37:34.900336 20695 log.cc:824] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:34.902525 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c: No bootstrap required, opened a new log
I20260927 16:37:34.905066 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:34.892025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35825 (local address 127.0.0.1:44769)
0927 16:37:34.892493 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:34.892507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:34.892542 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:34.893327 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:34.893346 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:34.893706 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:34.894046 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:34.894892 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:34.895901 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:34.896774 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:34.897296 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:34.897465 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:34.900292 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:34.900297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:34.900310 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:34.900340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:34.902227 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:34.902809 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:34.902814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:34.902818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:34.902929 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:34.903285 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:34.903287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:34.903288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:34.903647 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:34.903811 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:34.904115 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 16:37:34.904363 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":182,"threads_started":1}
I20260927 16:37:34.906661 20695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260927 16:37:34.906919 20695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:34.906991 20695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f868bb51f4495196ec086df2bbc33c, State: Initialized, Role: FOLLOWER
I20260927 16:37:34.907382 20695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260927 16:37:34.907538 20695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:34.907611 20695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:34.907742 20695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:34.908196 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:34.908752 20695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260927 16:37:34.909078 20695 leader_election.cc:302] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f868bb51f4495196ec086df2bbc33c; no voters:
I20260927 16:37:34.909386 20695 leader_election.cc:288] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:34.909480 20701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:34.909651 20701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [term 1 LEADER]: Becoming Leader. State: Replica: 70f868bb51f4495196ec086df2bbc33c, State: Running, Role: LEADER
I20260927 16:37:34.909934 20701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } }
I20260927 16:37:34.910658 20695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:34.911087 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f868bb51f4495196ec086df2bbc33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } } }
I20260927 16:37:34.911207 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:34.911404 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f868bb51f4495196ec086df2bbc33c. Latest consensus state: current_term: 1 leader_uuid: "70f868bb51f4495196ec086df2bbc33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f868bb51f4495196ec086df2bbc33c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44769 } } }
I20260927 16:37:34.911504 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:34.911634 20709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:34.914894 20709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:34.921303 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:34.910328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45749 (local address 127.0.0.1:44769)
0927 16:37:34.910450 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:34.910455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:34.911803 (+ 1348us) server_negotiation.cc:407] Connection header received
0927 16:37:34.911992 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:34.911996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:34.912040 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:34.912087 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:34.913168 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:34.913932 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:34.915135 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:34.915471 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:34.915550 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:34.917428 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:34.917431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:34.917434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:34.917459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:34.919226 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:34.919841 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:34.919845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:34.919847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:34.919944 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:34.920346 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:34.920350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:34.920351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:34.920568 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:34.920720 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:34.921021 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 16:37:34.921126 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260927 16:37:34.921651 20709 catalog_manager.cc:1347] Generated new cluster ID: beffc89669c94e2b8876c9408228c615
I20260927 16:37:34.921734 20709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:34.932760 20709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:34.933655 20709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:34.942761 20709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f868bb51f4495196ec086df2bbc33c: Generated new TSK 0
I20260927 16:37:34.943615 20709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:34.952687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20627
W20260927 16:37:34.958227 20612 connection.cc:535] client connection to 127.0.0.1:44769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44769): Connection reset by peer (error 104)
2026-09-27T16:37:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:35 dist-test-slave-t27c krb5kdc[20726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:35 dist-test-slave-t27c krb5kdc[20726](info): set up 2 sockets
Sep 27 16:37:35 dist-test-slave-t27c krb5kdc[20726](info): commencing operation
krb5kdc: starting...
W20260927 16:37:35.990309 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:36 dist-test-slave-t27c krb5kdc[20726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:36.109483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:36.222291 20739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:36.222630 20739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:36.222723 20739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:36.226774 20739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:36.226862 20739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:36.226917 20739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:36.226971 20739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:36.227012 20739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:36.232223 20739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:36.233292 20739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:36.234359 20739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:36.242914 20744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:36.243016 20739 server_base.cc:1034] running on GCE node
W20260927 16:37:36.242952 20745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:36.242928 20747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:36.243754 20739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:36.244918 20739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:36.246105 20739 hybrid_clock.cc:648] HybridClock initialized: now 1790527056246086 us; error 32 us; skew 500 ppm
Sep 27 16:37:36 dist-test-slave-t27c krb5kdc[20726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:36.248708 20739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:36.249544 20739 webserver.cc:460] Webserver started at http://127.0.0.1:36601/ using document root <none> and password file <none>
I20260927 16:37:36.250100 20739 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:36.250208 20739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:36.250427 20739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:36.252408 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b21b5c95f0aa4fa383ba1d95b044e755"
format_stamp: "Formatted at 2026-09-27 16:37:36 on dist-test-slave-t27c"
server_key: "5e125a5b1727355b1882f9cfaf79cd87"
server_key_iv: "ba6be65ae54b84c1a2bd722835c5fa6a"
server_key_version: "encryptionkey@0"
I20260927 16:37:36.252967 20739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b21b5c95f0aa4fa383ba1d95b044e755"
format_stamp: "Formatted at 2026-09-27 16:37:36 on dist-test-slave-t27c"
server_key: "5e125a5b1727355b1882f9cfaf79cd87"
server_key_iv: "ba6be65ae54b84c1a2bd722835c5fa6a"
server_key_version: "encryptionkey@0"
I20260927 16:37:36.256606 20739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:37:36.258970 20754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:36.260123 20739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:36.260258 20739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b21b5c95f0aa4fa383ba1d95b044e755"
format_stamp: "Formatted at 2026-09-27 16:37:36 on dist-test-slave-t27c"
server_key: "5e125a5b1727355b1882f9cfaf79cd87"
server_key_iv: "ba6be65ae54b84c1a2bd722835c5fa6a"
server_key_version: "encryptionkey@0"
I20260927 16:37:36.260368 20739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:36.282953 20739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:36.285908 20739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:36.286114 20739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:36.293107 20739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36947
I20260927 16:37:36.293133 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36947 every 8 connection(s)
I20260927 16:37:36.294023 20739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:36.295148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20739
I20260927 16:37:36.295297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:36.295578 392 external_mini_cluster.cc:1428] Setting key 743870713d0d1f7132a8d3e58553e7ad
I20260927 16:37:36.296975 20807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:36.303009 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755: Bootstrap starting.
I20260927 16:37:36.304572 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:36.297003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57783 (local address 127.0.0.1:36947)
0927 16:37:36.297444 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:36.297458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:36.297494 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:37:36.298115 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:36.298130 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:36.298561 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:36.298900 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:36.299321 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:36.300303 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:36.301264 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:36.301737 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:36.301912 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:36.302881 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:36.302886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:36.302900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:36.303361 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:36.303416 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:36.303440 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:36.303581 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:36.303730 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:37:36.303871 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"spinlock_wait_cycles":23424,"thread_start_us":229,"threads_started":1}
I20260927 16:37:36.305420 20807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:36.306291 20807 log.cc:824] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:36.307664 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:36.308347 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755: No bootstrap required, opened a new log
I20260927 16:37:36.312521 20807 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260927 16:37:36.312772 20807 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:36.312832 20807 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21b5c95f0aa4fa383ba1d95b044e755, State: Initialized, Role: FOLLOWER
I20260927 16:37:36.313326 20807 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260927 16:37:36.313519 20807 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:36.313603 20807 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:36.313715 20807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:36.314219 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:36.309808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55791 (local address 127.0.0.1:36947)
0927 16:37:36.309933 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:36.309938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:36.310127 (+ 189us) server_negotiation.cc:407] Connection header received
0927 16:37:36.310300 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:36.310303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:36.310338 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:36.310402 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:36.310736 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:36.311295 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:36.312415 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:36.312654 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:36.312733 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:36.313687 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:36.313691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:36.313694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:36.313705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:36.313744 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:36.313757 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:36.313851 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:36.314020 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 16:37:36.314059 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:37:36.314822 20807 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260927 16:37:36.315176 20807 leader_election.cc:302] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21b5c95f0aa4fa383ba1d95b044e755; no voters:
I20260927 16:37:36.315515 20807 leader_election.cc:288] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:36.315799 20817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:36.316016 20817 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [term 1 LEADER]: Becoming Leader. State: Replica: b21b5c95f0aa4fa383ba1d95b044e755, State: Running, Role: LEADER
I20260927 16:37:36.316358 20817 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260927 16:37:36.316937 20807 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:36.318231 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21b5c95f0aa4fa383ba1d95b044e755. Latest consensus state: current_term: 1 leader_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260927 16:37:36.318367 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:36.318639 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b5c95f0aa4fa383ba1d95b044e755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260927 16:37:36.318733 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:36.318882 20826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:36.322126 20826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:36.327881 20826 catalog_manager.cc:1347] Generated new cluster ID: 83557b477b174458993ce145f26cec93
I20260927 16:37:36.327951 20826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:36.350642 20826 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:36.351557 20826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:36.359198 20826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21b5c95f0aa4fa383ba1d95b044e755: Generated new TSK 0
I20260927 16:37:36.359951 20826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:36.372473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20739
W20260927 16:37:36.377288 20724 connection.cc:535] client connection to 127.0.0.1:36947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36947): Connection reset by peer (error 104)
2026-09-27T16:37:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:36 dist-test-slave-t27c krb5kdc[20838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:36 dist-test-slave-t27c krb5kdc[20838](info): set up 2 sockets
Sep 27 16:37:36 dist-test-slave-t27c krb5kdc[20838](info): commencing operation
krb5kdc: starting...
W20260927 16:37:37.414390 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:37 dist-test-slave-t27c krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:37.537213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:37.648093 20851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:37.648423 20851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:37.648524 20851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:37.652666 20851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:37.652756 20851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:37.652791 20851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:37.652848 20851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:37.652887 20851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:37.658330 20851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:37.659444 20851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:37.660485 20851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:37.668922 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:37.668968 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:37.669130 20851 server_base.cc:1034] running on GCE node
W20260927 16:37:37.668917 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:37.669847 20851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:37.671033 20851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:37.672228 20851 hybrid_clock.cc:648] HybridClock initialized: now 1790527057672196 us; error 46 us; skew 500 ppm
Sep 27 16:37:37 dist-test-slave-t27c krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:37.675113 20851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:37.676026 20851 webserver.cc:460] Webserver started at http://127.0.0.1:43589/ using document root <none> and password file <none>
I20260927 16:37:37.676607 20851 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:37.676707 20851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:37.676968 20851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:37.679163 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "10e1c99dc4d643c88a630cf32cf0ffba"
format_stamp: "Formatted at 2026-09-27 16:37:37 on dist-test-slave-t27c"
server_key: "5c626c79836b94b4cfa713e421df1301"
server_key_iv: "6d58f3076e9517db775194c3ef4f7689"
server_key_version: "encryptionkey@0"
I20260927 16:37:37.679747 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "10e1c99dc4d643c88a630cf32cf0ffba"
format_stamp: "Formatted at 2026-09-27 16:37:37 on dist-test-slave-t27c"
server_key: "5c626c79836b94b4cfa713e421df1301"
server_key_iv: "6d58f3076e9517db775194c3ef4f7689"
server_key_version: "encryptionkey@0"
I20260927 16:37:37.683555 20851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:37:37.685956 20866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:37.687242 20851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:37:37.687388 20851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "10e1c99dc4d643c88a630cf32cf0ffba"
format_stamp: "Formatted at 2026-09-27 16:37:37 on dist-test-slave-t27c"
server_key: "5c626c79836b94b4cfa713e421df1301"
server_key_iv: "6d58f3076e9517db775194c3ef4f7689"
server_key_version: "encryptionkey@0"
I20260927 16:37:37.687506 20851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:37.692329 20851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:37.695708 20851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:37.695909 20851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:37.702663 20851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38535
I20260927 16:37:37.702678 20918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38535 every 8 connection(s)
I20260927 16:37:37.703635 20851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:37.706398 20919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:37.711733 20919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba: Bootstrap starting.
I20260927 16:37:37.713200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20851
I20260927 16:37:37.713382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:37.713644 392 external_mini_cluster.cc:1428] Setting key 76484653a941be9ee58d39ce0bf5392b
I20260927 16:37:37.714298 20919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:37.715200 20919 log.cc:824] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:37.717305 20919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba: No bootstrap required, opened a new log
I20260927 16:37:37.718992 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:37.714761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42537 (local address 127.0.0.1:38535)
0927 16:37:37.715215 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:37.715229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:37.715263 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:37:37.716136 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:37.716159 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:37.716458 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:37.716830 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:37.717687 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:37.717692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:37.717705 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:37.717715 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:37.717777 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:37.717797 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:37.717916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:37.718022 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:37:37.718229 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260927 16:37:37.721720 20919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } }
I20260927 16:37:37.722036 20919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:37.722097 20919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e1c99dc4d643c88a630cf32cf0ffba, State: Initialized, Role: FOLLOWER
I20260927 16:37:37.722405 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:37.722673 20919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } }
I20260927 16:37:37.722831 20919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:37.722899 20919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:37.723002 20919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:37.724161 20919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } }
I20260927 16:37:37.724570 20919 leader_election.cc:302] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10e1c99dc4d643c88a630cf32cf0ffba; no voters:
I20260927 16:37:37.724933 20919 leader_election.cc:288] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:37.725023 20929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:37.725297 20929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [term 1 LEADER]: Becoming Leader. State: Replica: 10e1c99dc4d643c88a630cf32cf0ffba, State: Running, Role: LEADER
I20260927 16:37:37.725590 20929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } }
I20260927 16:37:37.726415 20919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:37.726567 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:37.724578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47185 (local address 127.0.0.1:38535)
0927 16:37:37.724711 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:37.724715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:37.724969 (+ 254us) server_negotiation.cc:407] Connection header received
0927 16:37:37.725125 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:37.725128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:37.725166 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:37.725251 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:37.726028 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:37.726031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:37.726034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:37.726045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:37.726072 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:37.726084 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:37.726165 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:37.726341 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 16:37:37.726382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:37.727469 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } } }
I20260927 16:37:37.727490 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e1c99dc4d643c88a630cf32cf0ffba. Latest consensus state: current_term: 1 leader_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e1c99dc4d643c88a630cf32cf0ffba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38535 } } }
I20260927 16:37:37.727648 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:37.727664 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:37.728021 20938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:37.730938 20938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:37.735836 20938 catalog_manager.cc:1347] Generated new cluster ID: bfcbc378f96448a0adde80022109026c
I20260927 16:37:37.735898 20938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:37.745944 20938 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:37.746310 20938 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:37.753172 20938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10e1c99dc4d643c88a630cf32cf0ffba: Generated new TSK 0
I20260927 16:37:37.753883 20938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:37.757134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20851
2026-09-27T16:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:37 dist-test-slave-t27c krb5kdc[20950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:37 dist-test-slave-t27c krb5kdc[20950](info): set up 2 sockets
Sep 27 16:37:37 dist-test-slave-t27c krb5kdc[20950](info): commencing operation
krb5kdc: starting...
W20260927 16:37:38.794299 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:38 dist-test-slave-t27c krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:38.915012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:39.027505 20963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:39.027809 20963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:39.027885 20963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:39.031941 20963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:39.032030 20963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:39.032068 20963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:39.032128 20963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:39.032166 20963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:39.038077 20963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:39.039372 20963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:39.040488 20963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:39.049268 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:39.049309 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:39.049414 20963 server_base.cc:1034] running on GCE node
W20260927 16:37:39.049360 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:39.050137 20963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:39.051371 20963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:39.052563 20963 hybrid_clock.cc:648] HybridClock initialized: now 1790527059052540 us; error 35 us; skew 500 ppm
Sep 27 16:37:39 dist-test-slave-t27c krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:39.061800 20963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:39.062768 20963 webserver.cc:460] Webserver started at http://127.0.0.1:34437/ using document root <none> and password file <none>
I20260927 16:37:39.063359 20963 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:39.063453 20963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:39.063685 20963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:39.065845 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "cf2779e901334c50bba327f9bdc4a9b7"
format_stamp: "Formatted at 2026-09-27 16:37:39 on dist-test-slave-t27c"
server_key: "449f66192b7e15f34748ac3bbfaa99cc"
server_key_iv: "715759c0724c7cd2a4db712ff72335ca"
server_key_version: "encryptionkey@0"
I20260927 16:37:39.066447 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf2779e901334c50bba327f9bdc4a9b7"
format_stamp: "Formatted at 2026-09-27 16:37:39 on dist-test-slave-t27c"
server_key: "449f66192b7e15f34748ac3bbfaa99cc"
server_key_iv: "715759c0724c7cd2a4db712ff72335ca"
server_key_version: "encryptionkey@0"
I20260927 16:37:39.070250 20963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:37:39.072609 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:39.073846 20963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:39.073988 20963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "cf2779e901334c50bba327f9bdc4a9b7"
format_stamp: "Formatted at 2026-09-27 16:37:39 on dist-test-slave-t27c"
server_key: "449f66192b7e15f34748ac3bbfaa99cc"
server_key_iv: "715759c0724c7cd2a4db712ff72335ca"
server_key_version: "encryptionkey@0"
I20260927 16:37:39.074105 20963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:39.093050 20963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:39.096050 20963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:39.096259 20963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:39.103699 20963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43379
I20260927 16:37:39.103729 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43379 every 8 connection(s)
I20260927 16:37:39.104677 20963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:39.107439 21031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:39.111608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 20963
I20260927 16:37:39.111758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:39.112047 392 external_mini_cluster.cc:1428] Setting key 6eb54c3301543fd96d6286119580b3e6
I20260927 16:37:39.113508 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: Bootstrap starting.
I20260927 16:37:39.115849 21031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:39.116613 21031 log.cc:824] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:39.118675 21031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: No bootstrap required, opened a new log
Sep 27 16:37:39 dist-test-slave-t27c krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:37:39.122892 21031 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260927 16:37:39.123162 21031 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:39.123240 21031 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2779e901334c50bba327f9bdc4a9b7, State: Initialized, Role: FOLLOWER
I20260927 16:37:39.123832 21031 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260927 16:37:39.124063 21031 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:39.124145 21031 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:39.124250 21031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:39.125391 21031 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260927 16:37:39.125741 21031 leader_election.cc:302] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2779e901334c50bba327f9bdc4a9b7; no voters:
I20260927 16:37:39.126091 21031 leader_election.cc:288] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:39.126215 21036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:39.126426 21036 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [term 1 LEADER]: Becoming Leader. State: Replica: cf2779e901334c50bba327f9bdc4a9b7, State: Running, Role: LEADER
I20260927 16:37:39.126519 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:39.113471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47639 (local address 127.0.0.1:43379)
0927 16:37:39.113892 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:39.113906 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:39.113941 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:39.114617 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:39.114635 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:39.115005 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:39.115344 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:39.116173 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:39.117222 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:39.118113 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:39.118667 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:39.118858 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:39.121336 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:39.121342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:39.121356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:39.121384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:39.123288 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:39.123821 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:39.123825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:39.123830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:39.123954 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:39.124305 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:39.124308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:39.124309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:39.124734 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:39.124894 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:39.125185 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 16:37:39.125470 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260927 16:37:39.126739 21036 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } }
I20260927 16:37:39.127360 21031 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:39.127890 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2779e901334c50bba327f9bdc4a9b7. Latest consensus state: current_term: 1 leader_uuid: "cf2779e901334c50bba327f9bdc4a9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } } }
I20260927 16:37:39.128001 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:39.128437 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf2779e901334c50bba327f9bdc4a9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2779e901334c50bba327f9bdc4a9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43379 } } }
I20260927 16:37:39.128602 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:39.129454 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 16:37:39.133176 21051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:37:39.133266 21051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:37:39.133352 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:39.134061 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:39.139945 21041 catalog_manager.cc:1347] Generated new cluster ID: 4fdc80094127461a867f53786660d784
I20260927 16:37:39.140007 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:39.140753 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:39.131755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36907 (local address 127.0.0.1:43379)
0927 16:37:39.131878 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:39.131883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:39.132351 (+ 468us) server_negotiation.cc:407] Connection header received
0927 16:37:39.132602 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:39.132606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:39.132655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:39.132753 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:39.133515 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:39.134036 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:39.135154 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:39.135503 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:39.135646 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:39.136990 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:39.136994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:39.136997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:39.137020 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:39.138981 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:39.139483 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:39.139486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:39.139488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:39.139576 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:39.139894 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:39.139897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:39.139899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:39.140114 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:39.140249 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:39.140487 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:37:39.140600 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:37:39.161234 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:39.162329 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:39.166905 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2779e901334c50bba327f9bdc4a9b7: Generated new TSK 0
I20260927 16:37:39.167573 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:39.174300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 20963
2026-09-27T16:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:39 dist-test-slave-t27c krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:39 dist-test-slave-t27c krb5kdc[21062](info): set up 2 sockets
Sep 27 16:37:39 dist-test-slave-t27c krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260927 16:37:40.222500 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:40 dist-test-slave-t27c krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:40.341483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:40.456806 21075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:40.457103 21075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:40.457185 21075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:40.461264 21075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:40.461360 21075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:40.461400 21075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:40.461457 21075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:40.461495 21075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:40.467056 21075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:40.468182 21075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:40.469220 21075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:40.477938 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:40.478132 21075 server_base.cc:1034] running on GCE node
W20260927 16:37:40.477958 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:40.477958 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:40.478941 21075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:40.480366 21075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:40.481565 21075 hybrid_clock.cc:648] HybridClock initialized: now 1790527060481535 us; error 51 us; skew 500 ppm
Sep 27 16:37:40 dist-test-slave-t27c krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:40.484563 21075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:40.485495 21075 webserver.cc:460] Webserver started at http://127.0.0.1:34421/ using document root <none> and password file <none>
I20260927 16:37:40.486124 21075 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:40.486239 21075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:40.486482 21075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:40.488638 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "e7088b8fe85943b3a733c74ad901b706"
format_stamp: "Formatted at 2026-09-27 16:37:40 on dist-test-slave-t27c"
server_key: "a7979b4d17b8c39b6f9d32dc77a3995e"
server_key_iv: "841a49a8e42561dc36632af84c0e5f87"
server_key_version: "encryptionkey@0"
I20260927 16:37:40.489207 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7088b8fe85943b3a733c74ad901b706"
format_stamp: "Formatted at 2026-09-27 16:37:40 on dist-test-slave-t27c"
server_key: "a7979b4d17b8c39b6f9d32dc77a3995e"
server_key_iv: "841a49a8e42561dc36632af84c0e5f87"
server_key_version: "encryptionkey@0"
I20260927 16:37:40.492957 21075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 16:37:40.495396 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:40.496600 21075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:40.496745 21075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "e7088b8fe85943b3a733c74ad901b706"
format_stamp: "Formatted at 2026-09-27 16:37:40 on dist-test-slave-t27c"
server_key: "a7979b4d17b8c39b6f9d32dc77a3995e"
server_key_iv: "841a49a8e42561dc36632af84c0e5f87"
server_key_version: "encryptionkey@0"
I20260927 16:37:40.496873 21075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:40.508000 21075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:40.511274 21075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:40.511494 21075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:40.518306 21075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33865
I20260927 16:37:40.518380 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33865 every 8 connection(s)
I20260927 16:37:40.519382 21075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:40.522405 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:40.527752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21075
I20260927 16:37:40.527920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:40.528203 392 external_mini_cluster.cc:1428] Setting key 8dbdb1673d92e9b145b718f65d89b374
I20260927 16:37:40.528244 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706: Bootstrap starting.
I20260927 16:37:40.530790 21143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:40.531642 21143 log.cc:824] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:40.533661 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706: No bootstrap required, opened a new log
I20260927 16:37:40.536921 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:40.529508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46577 (local address 127.0.0.1:33865)
0927 16:37:40.529926 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:40.529942 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:40.529979 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:37:40.530646 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:40.530664 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:40.531050 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:40.531333 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:40.531646 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:40.532619 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:40.533838 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:40.534364 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:40.534478 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:40.535286 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:40.535291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:40.535302 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:40.535688 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:40.535739 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:40.535759 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:40.535834 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:40.536001 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:37:40.536216 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
I20260927 16:37:40.537796 21143 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } }
I20260927 16:37:40.538082 21143 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:40.538162 21143 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7088b8fe85943b3a733c74ad901b706, State: Initialized, Role: FOLLOWER
I20260927 16:37:40.538635 21143 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } }
I20260927 16:37:40.538800 21143 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:40.538872 21143 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:40.538962 21143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:40.539904 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:40.540078 21143 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } }
I20260927 16:37:40.540408 21143 leader_election.cc:302] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7088b8fe85943b3a733c74ad901b706; no voters:
I20260927 16:37:40.540740 21143 leader_election.cc:288] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:40.540804 21148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:40.541004 21148 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [term 1 LEADER]: Becoming Leader. State: Replica: e7088b8fe85943b3a733c74ad901b706, State: Running, Role: LEADER
I20260927 16:37:40.541343 21148 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } }
I20260927 16:37:40.542001 21143 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:40.542996 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7088b8fe85943b3a733c74ad901b706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } } }
I20260927 16:37:40.543118 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:40.543488 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7088b8fe85943b3a733c74ad901b706. Latest consensus state: current_term: 1 leader_uuid: "e7088b8fe85943b3a733c74ad901b706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7088b8fe85943b3a733c74ad901b706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33865 } } }
I20260927 16:37:40.543579 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:40.543816 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:40.546974 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:40.548556 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:40.542941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44465 (local address 127.0.0.1:33865)
0927 16:37:40.543054 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:40.543059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:40.544214 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 16:37:40.544420 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:40.544424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:40.544472 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:40.544556 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:40.544852 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:40.545591 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:40.546778 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:40.547110 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:40.547270 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:40.548106 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:40.548110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:40.548112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:40.548127 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:40.548157 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:40.548170 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:40.548249 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:40.548375 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 16:37:40.548428 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:37:40.553409 21161 catalog_manager.cc:1347] Generated new cluster ID: 99d63088e7064f04b7e27d46f5e07c8c
I20260927 16:37:40.553476 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:40.587399 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:40.588402 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:40.600956 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7088b8fe85943b3a733c74ad901b706: Generated new TSK 0
I20260927 16:37:40.601800 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:40.610810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21075
2026-09-27T16:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:40 dist-test-slave-t27c krb5kdc[21174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:40 dist-test-slave-t27c krb5kdc[21174](info): set up 2 sockets
Sep 27 16:37:40 dist-test-slave-t27c krb5kdc[21174](info): commencing operation
krb5kdc: starting...
W20260927 16:37:41.642541 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:41 dist-test-slave-t27c krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:41.763393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:41.873399 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:41.873656 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:41.873695 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:41.877651 21187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:41.877722 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:41.877741 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:41.877759 21187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:41.877774 21187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:41.883285 21187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:41.884403 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:41.885402 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:41.893852 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:41.893862 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:41.893867 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:41.894197 21187 server_base.cc:1034] running on GCE node
I20260927 16:37:41.894809 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:41.895947 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:41.897131 21187 hybrid_clock.cc:648] HybridClock initialized: now 1790527061897118 us; error 29 us; skew 500 ppm
Sep 27 16:37:41 dist-test-slave-t27c krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:41.899837 21187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:41.900698 21187 webserver.cc:460] Webserver started at http://127.0.0.1:33991/ using document root <none> and password file <none>
I20260927 16:37:41.901216 21187 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:41.901274 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:41.901436 21187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:41.903442 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b19e87b9b45a490aac6ee276043ea439"
format_stamp: "Formatted at 2026-09-27 16:37:41 on dist-test-slave-t27c"
server_key: "72e3fba7b24ae6cc93ce0cbc6a7c1293"
server_key_iv: "4b7fd352bcb5fb423b34dd8bb797fb7f"
server_key_version: "encryptionkey@0"
I20260927 16:37:41.903955 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b19e87b9b45a490aac6ee276043ea439"
format_stamp: "Formatted at 2026-09-27 16:37:41 on dist-test-slave-t27c"
server_key: "72e3fba7b24ae6cc93ce0cbc6a7c1293"
server_key_iv: "4b7fd352bcb5fb423b34dd8bb797fb7f"
server_key_version: "encryptionkey@0"
I20260927 16:37:41.907461 21187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:41.909785 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:41.911029 21187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:37:41.911134 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b19e87b9b45a490aac6ee276043ea439"
format_stamp: "Formatted at 2026-09-27 16:37:41 on dist-test-slave-t27c"
server_key: "72e3fba7b24ae6cc93ce0cbc6a7c1293"
server_key_iv: "4b7fd352bcb5fb423b34dd8bb797fb7f"
server_key_version: "encryptionkey@0"
I20260927 16:37:41.911217 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:41.915619 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:41.918619 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:41.918781 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:41.926046 21187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33009
I20260927 16:37:41.926060 21254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33009 every 8 connection(s)
I20260927 16:37:41.927166 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:41.929616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21187
I20260927 16:37:41.929755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:41.930094 392 external_mini_cluster.cc:1428] Setting key 58c9d18d9860cce6b9e42696405638b9
I20260927 16:37:41.930454 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:41.935163 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:41.931477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40791 (local address 127.0.0.1:33009)
0927 16:37:41.931869 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:41.931890 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:41.931925 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:41.932575 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:41.932591 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:41.932870 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:41.933141 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:41.933968 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:41.933973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:41.933984 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:41.933994 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:41.934045 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:41.934064 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:41.934215 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:41.934316 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 16:37:41.934487 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260927 16:37:41.937077 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439: Bootstrap starting.
I20260927 16:37:41.937882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:41.939460 21255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:41.940289 21255 log.cc:824] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:41.941717 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:41.939975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52881 (local address 127.0.0.1:33009)
0927 16:37:41.940098 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:41.940102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:41.940334 (+ 232us) server_negotiation.cc:407] Connection header received
0927 16:37:41.940456 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:41.940462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:41.940502 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:41.940566 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:41.941319 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:41.941321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:41.941323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:41.941333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:41.941354 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:41.941370 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:41.941435 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:41.941538 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 16:37:41.941568 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:37:41.942482 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439: No bootstrap required, opened a new log
I20260927 16:37:41.945761 21255 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260927 16:37:41.945963 21255 raft_consensus.cc:374] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:41.946048 21255 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19e87b9b45a490aac6ee276043ea439, State: Initialized, Role: FOLLOWER
I20260927 16:37:41.946480 21255 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260927 16:37:41.946619 21255 raft_consensus.cc:388] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:41.946681 21255 raft_consensus.cc:482] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:41.946779 21255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:41.947808 21255 raft_consensus.cc:504] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260927 16:37:41.948129 21255 leader_election.cc:302] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b19e87b9b45a490aac6ee276043ea439; no voters:
I20260927 16:37:41.948422 21255 leader_election.cc:288] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:41.948529 21265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:41.948778 21265 raft_consensus.cc:686] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [term 1 LEADER]: Becoming Leader. State: Replica: b19e87b9b45a490aac6ee276043ea439, State: Running, Role: LEADER
I20260927 16:37:41.949110 21265 consensus_queue.cc:237] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } }
I20260927 16:37:41.949595 21255 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:41.950680 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b19e87b9b45a490aac6ee276043ea439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } } }
I20260927 16:37:41.950974 21267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b19e87b9b45a490aac6ee276043ea439. Latest consensus state: current_term: 1 leader_uuid: "b19e87b9b45a490aac6ee276043ea439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19e87b9b45a490aac6ee276043ea439" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33009 } } }
I20260927 16:37:41.951087 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:41.951167 21267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:41.951983 21276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:41.954650 21276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:41.960048 21276 catalog_manager.cc:1347] Generated new cluster ID: 1cd375f40d1644d795046c9130e4d9a3
I20260927 16:37:41.960132 21276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:41.977133 21276 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:41.977474 21276 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:41.983619 21276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b19e87b9b45a490aac6ee276043ea439: Generated new TSK 0
I20260927 16:37:41.984265 21276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:41.996646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21187
2026-09-27T16:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:42 dist-test-slave-t27c krb5kdc[21286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:42 dist-test-slave-t27c krb5kdc[21286](info): set up 2 sockets
Sep 27 16:37:42 dist-test-slave-t27c krb5kdc[21286](info): commencing operation
krb5kdc: starting...
W20260927 16:37:43.054258 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:43.175650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:43.288731 21299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:43.289021 21299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:43.289067 21299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:43.293128 21299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:43.293186 21299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:43.293206 21299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:43.293223 21299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:43.293238 21299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:43.298441 21299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:43.299434 21299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:43.300390 21299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:43.308630 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:43.308712 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:43.308753 21299 server_base.cc:1034] running on GCE node
W20260927 16:37:43.308653 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:43.309365 21299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:43.310482 21299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:43.311640 21299 hybrid_clock.cc:648] HybridClock initialized: now 1790527063311615 us; error 45 us; skew 500 ppm
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:43.314364 21299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:43.315253 21299 webserver.cc:460] Webserver started at http://127.0.0.1:33117/ using document root <none> and password file <none>
I20260927 16:37:43.315809 21299 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:43.315872 21299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:43.316037 21299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:43.318037 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "bfa5cf283c744bfa8e20acf6e359ba31"
format_stamp: "Formatted at 2026-09-27 16:37:43 on dist-test-slave-t27c"
server_key: "3c83023e2c5e73dac48a350ae50960c2"
server_key_iv: "d088987771f4f1295e5f06b3dbe2828a"
server_key_version: "encryptionkey@0"
I20260927 16:37:43.318706 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfa5cf283c744bfa8e20acf6e359ba31"
format_stamp: "Formatted at 2026-09-27 16:37:43 on dist-test-slave-t27c"
server_key: "3c83023e2c5e73dac48a350ae50960c2"
server_key_iv: "d088987771f4f1295e5f06b3dbe2828a"
server_key_version: "encryptionkey@0"
I20260927 16:37:43.322604 21299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:37:43.325174 21314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:43.326485 21299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:43.326623 21299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "bfa5cf283c744bfa8e20acf6e359ba31"
format_stamp: "Formatted at 2026-09-27 16:37:43 on dist-test-slave-t27c"
server_key: "3c83023e2c5e73dac48a350ae50960c2"
server_key_iv: "d088987771f4f1295e5f06b3dbe2828a"
server_key_version: "encryptionkey@0"
I20260927 16:37:43.326746 21299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:43.354485 21299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:43.357604 21299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:43.357827 21299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:43.365446 21299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37113
I20260927 16:37:43.365465 21366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37113 every 8 connection(s)
I20260927 16:37:43.366478 21299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:43.369215 21367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:43.371866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21299
I20260927 16:37:43.372030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:43.372319 392 external_mini_cluster.cc:1428] Setting key 16a92814067459f0eea01f20cf234ae8
I20260927 16:37:43.375447 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31: Bootstrap starting.
I20260927 16:37:43.377561 21367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:43.378374 21367 log.cc:824] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:43.380366 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31: No bootstrap required, opened a new log
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:37:43.383692 21367 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } }
I20260927 16:37:43.383944 21367 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:43.384032 21367 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5cf283c744bfa8e20acf6e359ba31, State: Initialized, Role: FOLLOWER
I20260927 16:37:43.384500 21367 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } }
I20260927 16:37:43.384648 21367 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:43.384729 21367 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:43.384848 21367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:43.385720 21367 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } }
I20260927 16:37:43.386072 21367 leader_election.cc:302] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfa5cf283c744bfa8e20acf6e359ba31; no voters:
I20260927 16:37:43.386458 21367 leader_election.cc:288] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:43.386535 21372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:43.386773 21372 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [term 1 LEADER]: Becoming Leader. State: Replica: bfa5cf283c744bfa8e20acf6e359ba31, State: Running, Role: LEADER
I20260927 16:37:43.387156 21372 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } }
I20260927 16:37:43.387600 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:43.373791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:47665 (local address 127.0.0.1:37113)
0927 16:37:43.374281 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:43.374295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:43.374330 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:43.374933 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:43.374950 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:43.375382 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:43.375761 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:43.376533 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:43.377530 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:43.378502 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:43.379161 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:43.379350 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:43.381834 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:43.381840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:43.381853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:43.381880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:43.384181 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:43.384685 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:43.384691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:43.384704 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:43.384881 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:43.385287 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:43.385291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:43.385293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:43.385696 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:43.385932 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:43.386297 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 16:37:43.386579 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260927 16:37:43.387946 21367 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:43.388445 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } } }
I20260927 16:37:43.388501 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa5cf283c744bfa8e20acf6e359ba31. Latest consensus state: current_term: 1 leader_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa5cf283c744bfa8e20acf6e359ba31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37113 } } }
I20260927 16:37:43.388573 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:43.388599 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:43.389642 21376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:43.390935 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:43.393136 21376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:43.399565 21376 catalog_manager.cc:1347] Generated new cluster ID: 2cec5dcf48c14448ba9043972500fe2c
I20260927 16:37:43.399688 21376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:43.406751 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:43.394610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:53035 (local address 127.0.0.1:37113)
0927 16:37:43.394712 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:43.394716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:43.397498 (+ 2782us) server_negotiation.cc:407] Connection header received
0927 16:37:43.397668 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:43.397672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:43.397734 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:43.397824 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:43.398763 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:43.399589 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:43.400845 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:43.401197 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:43.401297 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:43.403147 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:43.403150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:43.403153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:43.403176 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:43.404859 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:43.405418 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:43.405421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:43.405423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:43.405500 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:43.405847 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:43.405849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:43.405851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:43.406041 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:43.406207 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:43.406492 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 16:37:43.406587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:37:43.421465 21376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:43.422806 21376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:43.431783 21376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfa5cf283c744bfa8e20acf6e359ba31: Generated new TSK 0
I20260927 16:37:43.432605 21376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:43.442317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21299
W20260927 16:37:43.447953 21284 connection.cc:535] client connection to 127.0.0.1:37113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37113): Connection reset by peer (error 104)
2026-09-27T16:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21398](info): set up 2 sockets
Sep 27 16:37:43 dist-test-slave-t27c krb5kdc[21398](info): commencing operation
krb5kdc: starting...
W20260927 16:37:44.510761 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:44 dist-test-slave-t27c krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:44.638329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:44.751888 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:44.752177 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:44.752220 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:44.756286 21411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:44.756352 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:44.756372 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:44.756389 21411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:44.756404 21411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:44.761790 21411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:44.763092 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:44.764268 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:44.773335 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:44.773419 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:44.773339 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:44.773730 21411 server_base.cc:1034] running on GCE node
I20260927 16:37:44.774390 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:44.775591 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:44.776805 21411 hybrid_clock.cc:648] HybridClock initialized: now 1790527064776769 us; error 49 us; skew 500 ppm
Sep 27 16:37:44 dist-test-slave-t27c krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:44.785495 21411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:44.786604 21411 webserver.cc:460] Webserver started at http://127.0.0.1:33543/ using document root <none> and password file <none>
I20260927 16:37:44.787225 21411 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:44.787320 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:44.787567 21411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:44.789711 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "a2a4b749be814e2983bcc929daa6b225"
format_stamp: "Formatted at 2026-09-27 16:37:44 on dist-test-slave-t27c"
server_key: "c9fa73d9733b7ba1231f549fa0269b9b"
server_key_iv: "1f2c0681080ef07780fc09dfca1aac00"
server_key_version: "encryptionkey@0"
I20260927 16:37:44.790362 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2a4b749be814e2983bcc929daa6b225"
format_stamp: "Formatted at 2026-09-27 16:37:44 on dist-test-slave-t27c"
server_key: "c9fa73d9733b7ba1231f549fa0269b9b"
server_key_iv: "1f2c0681080ef07780fc09dfca1aac00"
server_key_version: "encryptionkey@0"
I20260927 16:37:44.794147 21411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:37:44.797021 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:44.798442 21411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 16:37:44.798594 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "a2a4b749be814e2983bcc929daa6b225"
format_stamp: "Formatted at 2026-09-27 16:37:44 on dist-test-slave-t27c"
server_key: "c9fa73d9733b7ba1231f549fa0269b9b"
server_key_iv: "1f2c0681080ef07780fc09dfca1aac00"
server_key_version: "encryptionkey@0"
I20260927 16:37:44.798715 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:44.817298 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:44.820724 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:44.820943 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:44.828135 21411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33787
I20260927 16:37:44.828147 21478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33787 every 8 connection(s)
I20260927 16:37:44.829118 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:44.831902 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:44.835587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21411
I20260927 16:37:44.835772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:44.836102 392 external_mini_cluster.cc:1428] Setting key e3d059f35911518b09357eb58a0cb1b1
I20260927 16:37:44.837841 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225: Bootstrap starting.
I20260927 16:37:44.840314 21479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:44.841180 21479 log.cc:824] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:44.843675 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225: No bootstrap required, opened a new log
I20260927 16:37:44.845842 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:44.837483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:38689 (local address 127.0.0.1:33787)
0927 16:37:44.837911 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:44.837931 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:44.837992 (+ 61us) server_negotiation.cc:407] Connection header received
0927 16:37:44.838713 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:44.838732 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:44.839101 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:44.839408 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:44.839770 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:44.840862 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:44.842043 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:44.842632 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:44.842762 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:44.843743 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:44.843748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:44.843763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:44.844242 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:44.844300 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:44.844324 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:44.844437 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:44.844649 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 16:37:44.844833 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260927 16:37:44.847313 21479 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260927 16:37:44.847570 21479 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:44.847648 21479 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a4b749be814e2983bcc929daa6b225, State: Initialized, Role: FOLLOWER
I20260927 16:37:44.848136 21479 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260927 16:37:44.848304 21479 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:44.848397 21479 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:44.848498 21479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:44.849033 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:44.849524 21479 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260927 16:37:44.849856 21479 leader_election.cc:302] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a4b749be814e2983bcc929daa6b225; no voters:
I20260927 16:37:44.850232 21479 leader_election.cc:288] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:44.850281 21484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:44.850497 21484 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [term 1 LEADER]: Becoming Leader. State: Replica: a2a4b749be814e2983bcc929daa6b225, State: Running, Role: LEADER
I20260927 16:37:44.850845 21484 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260927 16:37:44.851549 21479 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:44.852213 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a4b749be814e2983bcc929daa6b225. Latest consensus state: current_term: 1 leader_uuid: "a2a4b749be814e2983bcc929daa6b225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260927 16:37:44.852414 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:44.852608 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a4b749be814e2983bcc929daa6b225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a4b749be814e2983bcc929daa6b225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260927 16:37:44.852747 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:44.852891 21498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:44.856189 21498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:44.858493 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:44.851165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:59609 (local address 127.0.0.1:33787)
0927 16:37:44.851294 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:44.851298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:44.853635 (+ 2337us) server_negotiation.cc:407] Connection header received
0927 16:37:44.853828 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:44.853834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:44.853892 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:44.853974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:44.854497 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:44.855242 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:44.856371 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:44.856704 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:44.856783 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:44.857908 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:44.857912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:44.857915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:44.857928 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:44.857957 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:44.857969 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:44.858060 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:44.858300 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 16:37:44.858339 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:44.861987 21498 catalog_manager.cc:1347] Generated new cluster ID: c5c84c5bb65b4d80aba8de7c9e72a0ac
I20260927 16:37:44.862061 21498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:44.875717 21498 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:44.876662 21498 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:44.885385 21498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a4b749be814e2983bcc929daa6b225: Generated new TSK 0
I20260927 16:37:44.886372 21498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:44.895694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21411
W20260927 16:37:44.901314 21396 connection.cc:535] client connection to 127.0.0.1:33787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33787): Connection reset by peer (error 104)
2026-09-27T16:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:44 dist-test-slave-t27c krb5kdc[21510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:44 dist-test-slave-t27c krb5kdc[21510](info): set up 2 sockets
Sep 27 16:37:44 dist-test-slave-t27c krb5kdc[21510](info): commencing operation
krb5kdc: starting...
W20260927 16:37:45.942515 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:46 dist-test-slave-t27c krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:46.064693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:46.179162 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:46.179597 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:46.179714 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:46.185622 21523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:46.185688 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:46.185707 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:46.185746 21523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:46.185778 21523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:46.191298 21523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:46.192382 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:46.193408 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:46.202114 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:46.202112 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:46.202318 21523 server_base.cc:1034] running on GCE node
W20260927 16:37:46.202525 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:46.203092 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:46.204346 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:46.205559 21523 hybrid_clock.cc:648] HybridClock initialized: now 1790527066205541 us; error 34 us; skew 500 ppm
Sep 27 16:37:46 dist-test-slave-t27c krb5kdc[21510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:46.208709 21523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:46.209645 21523 webserver.cc:460] Webserver started at http://127.0.0.1:45393/ using document root <none> and password file <none>
I20260927 16:37:46.210391 21523 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:46.210487 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:46.210705 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:46.212863 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b6605863dd0542f6974b7335148a9593"
format_stamp: "Formatted at 2026-09-27 16:37:46 on dist-test-slave-t27c"
server_key: "c1ced42283b9f5227a3b5976ddc09fb7"
server_key_iv: "958d79c85a7c95ac264c1dd5b1689e29"
server_key_version: "encryptionkey@0"
I20260927 16:37:46.213440 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6605863dd0542f6974b7335148a9593"
format_stamp: "Formatted at 2026-09-27 16:37:46 on dist-test-slave-t27c"
server_key: "c1ced42283b9f5227a3b5976ddc09fb7"
server_key_iv: "958d79c85a7c95ac264c1dd5b1689e29"
server_key_version: "encryptionkey@0"
I20260927 16:37:46.217567 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:37:46.220139 21538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:46.221378 21523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:46.221539 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b6605863dd0542f6974b7335148a9593"
format_stamp: "Formatted at 2026-09-27 16:37:46 on dist-test-slave-t27c"
server_key: "c1ced42283b9f5227a3b5976ddc09fb7"
server_key_iv: "958d79c85a7c95ac264c1dd5b1689e29"
server_key_version: "encryptionkey@0"
I20260927 16:37:46.221663 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:46.226275 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:46.229048 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:46.229241 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:46.236182 21523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33621
I20260927 16:37:46.236193 21590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33621 every 8 connection(s)
I20260927 16:37:46.237128 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:46.240058 21591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:46.240636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21523
I20260927 16:37:46.240782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:46.241106 392 external_mini_cluster.cc:1428] Setting key ebe4fe08a993df085011735cf7eab59d
I20260927 16:37:46.246763 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:46.242655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:57497 (local address 127.0.0.1:33621)
0927 16:37:46.243101 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:46.243118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:46.243160 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:37:46.243849 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:46.243869 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:46.244182 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:46.244491 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:46.245402 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:46.245408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:46.245426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:46.245437 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:46.245494 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:46.245513 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:46.245626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:46.245760 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 16:37:46.245939 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260927 16:37:46.247663 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593: Bootstrap starting.
I20260927 16:37:46.249760 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:46.250319 21591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:46.251207 21591 log.cc:824] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:46.253540 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593: No bootstrap required, opened a new log
I20260927 16:37:46.253707 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:46.252083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34885 (local address 127.0.0.1:33621)
0927 16:37:46.252209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:46.252213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:46.252388 (+ 175us) server_negotiation.cc:407] Connection header received
0927 16:37:46.252501 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:46.252503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:46.252530 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:46.252592 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:46.253280 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:46.253283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:46.253285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:46.253295 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:46.253318 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:46.253328 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:46.253392 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:46.253502 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 16:37:46.253533 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:37:46.257102 21591 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } }
I20260927 16:37:46.257349 21591 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:46.257407 21591 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6605863dd0542f6974b7335148a9593, State: Initialized, Role: FOLLOWER
I20260927 16:37:46.257804 21591 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } }
I20260927 16:37:46.257959 21591 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:46.258047 21591 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:46.258144 21591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:46.259063 21591 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } }
I20260927 16:37:46.259426 21591 leader_election.cc:302] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6605863dd0542f6974b7335148a9593; no voters:
I20260927 16:37:46.259728 21591 leader_election.cc:288] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:46.259840 21601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:46.260066 21601 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [term 1 LEADER]: Becoming Leader. State: Replica: b6605863dd0542f6974b7335148a9593, State: Running, Role: LEADER
I20260927 16:37:46.260390 21601 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } }
I20260927 16:37:46.260967 21591 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:46.262001 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6605863dd0542f6974b7335148a9593. Latest consensus state: current_term: 1 leader_uuid: "b6605863dd0542f6974b7335148a9593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } } }
I20260927 16:37:46.262131 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:46.262456 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6605863dd0542f6974b7335148a9593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6605863dd0542f6974b7335148a9593" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33621 } } }
I20260927 16:37:46.262562 21610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:46.262626 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:46.265656 21610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:46.271236 21610 catalog_manager.cc:1347] Generated new cluster ID: 3f1aabdcf2e34945815c5e6f6d6e37f2
I20260927 16:37:46.271318 21610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:46.300602 21610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:46.300980 21610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:46.304965 21610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6605863dd0542f6974b7335148a9593: Generated new TSK 0
I20260927 16:37:46.305625 21610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:46.314735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21523
2026-09-27T16:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:46 dist-test-slave-t27c krb5kdc[21622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:46 dist-test-slave-t27c krb5kdc[21622](info): set up 2 sockets
Sep 27 16:37:46 dist-test-slave-t27c krb5kdc[21622](info): commencing operation
krb5kdc: starting...
W20260927 16:37:47.366725 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:47.491478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:47.603216 21635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:47.603581 21635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:47.603669 21635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:47.607846 21635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:47.607966 21635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:47.608006 21635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:47.608045 21635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:47.608080 21635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:47.613575 21635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:47.614794 21635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:47.615857 21635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:47.624503 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:47.624512 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:47.624567 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:47.624516 21635 server_base.cc:1034] running on GCE node
I20260927 16:37:47.625386 21635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:47.626547 21635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:47.627736 21635 hybrid_clock.cc:648] HybridClock initialized: now 1790527067627722 us; error 34 us; skew 500 ppm
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:47.630630 21635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:47.631512 21635 webserver.cc:460] Webserver started at http://127.0.0.1:42265/ using document root <none> and password file <none>
I20260927 16:37:47.632082 21635 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:47.632144 21635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:47.632352 21635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:47.634629 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b541e7b166684a768d5a78e70ed2d71b"
format_stamp: "Formatted at 2026-09-27 16:37:47 on dist-test-slave-t27c"
server_key: "19bec87cfc06c98df026ab33b6cfe3a0"
server_key_iv: "23b187720ae0f7b1a759cc31d89d3231"
server_key_version: "encryptionkey@0"
I20260927 16:37:47.635191 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b541e7b166684a768d5a78e70ed2d71b"
format_stamp: "Formatted at 2026-09-27 16:37:47 on dist-test-slave-t27c"
server_key: "19bec87cfc06c98df026ab33b6cfe3a0"
server_key_iv: "23b187720ae0f7b1a759cc31d89d3231"
server_key_version: "encryptionkey@0"
I20260927 16:37:47.639153 21635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:37:47.641682 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:47.643128 21635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:37:47.643344 21635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b541e7b166684a768d5a78e70ed2d71b"
format_stamp: "Formatted at 2026-09-27 16:37:47 on dist-test-slave-t27c"
server_key: "19bec87cfc06c98df026ab33b6cfe3a0"
server_key_iv: "23b187720ae0f7b1a759cc31d89d3231"
server_key_version: "encryptionkey@0"
I20260927 16:37:47.643479 21635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:47.662647 21635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:47.666471 21635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:47.666684 21635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:47.674207 21635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44053
I20260927 16:37:47.674243 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44053 every 8 connection(s)
I20260927 16:37:47.675258 21635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:47.678009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21635
I20260927 16:37:47.678179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:47.678207 21703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:47.678498 392 external_mini_cluster.cc:1428] Setting key 3394e256d62ce3a7da0c81199ce5c98a
I20260927 16:37:47.685925 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b: Bootstrap starting.
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:37:47.689069 21703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:47.690339 21703 log.cc:824] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:47.692909 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b: No bootstrap required, opened a new log
I20260927 16:37:47.694599 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:47.679929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34717 (local address 127.0.0.1:44053)
0927 16:37:47.680364 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:47.680383 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:47.680426 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:37:47.681118 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:47.681141 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:47.681617 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:47.682007 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:47.682736 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:47.684286 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:47.685201 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:47.685913 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:47.686095 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:47.688870 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:47.688880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:47.688903 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:47.688949 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:47.691223 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:47.691743 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:47.691756 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:47.691762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:47.691902 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:47.692199 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:47.692202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:47.692203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:47.692702 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:47.692942 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:47.693436 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 16:37:47.693614 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260927 16:37:47.696506 21703 raft_consensus.cc:348] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } }
I20260927 16:37:47.696803 21703 raft_consensus.cc:374] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:47.696883 21703 raft_consensus.cc:729] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b541e7b166684a768d5a78e70ed2d71b, State: Initialized, Role: FOLLOWER
I20260927 16:37:47.697378 21703 consensus_queue.cc:260] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } }
I20260927 16:37:47.697538 21703 raft_consensus.cc:388] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:47.697608 21703 raft_consensus.cc:482] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:47.697755 21703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:47.697866 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:47.698808 21703 raft_consensus.cc:504] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } }
I20260927 16:37:47.699198 21703 leader_election.cc:302] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b541e7b166684a768d5a78e70ed2d71b; no voters:
I20260927 16:37:47.699560 21703 leader_election.cc:288] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:47.699645 21712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:47.699856 21712 raft_consensus.cc:686] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [term 1 LEADER]: Becoming Leader. State: Replica: b541e7b166684a768d5a78e70ed2d71b, State: Running, Role: LEADER
I20260927 16:37:47.700196 21712 consensus_queue.cc:237] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } }
I20260927 16:37:47.700891 21703 sys_catalog.cc:564] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:47.701375 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b541e7b166684a768d5a78e70ed2d71b. Latest consensus state: current_term: 1 leader_uuid: "b541e7b166684a768d5a78e70ed2d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } } }
I20260927 16:37:47.701488 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:47.701733 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b541e7b166684a768d5a78e70ed2d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b541e7b166684a768d5a78e70ed2d71b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44053 } } }
I20260927 16:37:47.701908 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:47.702244 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:47.705350 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:47.711284 21718 catalog_manager.cc:1347] Generated new cluster ID: 805f194c0be346e98201f82789137364
I20260927 16:37:47.711357 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:47.711469 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:47.699953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:49467 (local address 127.0.0.1:44053)
0927 16:37:47.700071 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:47.700075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:47.702465 (+ 2390us) server_negotiation.cc:407] Connection header received
0927 16:37:47.702656 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:47.702659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:47.702706 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:47.702759 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:47.703629 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:47.704401 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:47.705543 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:47.705859 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:47.705970 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:47.707657 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:47.707661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:47.707664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:47.707689 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:47.709385 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:47.709955 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:47.709958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:47.709960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:47.710053 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:47.710454 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:47.710458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:47.710460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:47.710671 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:47.710833 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:47.711132 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:37:47.711241 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:37:47.733220 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:47.734332 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:47.741964 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b541e7b166684a768d5a78e70ed2d71b: Generated new TSK 0
I20260927 16:37:47.742794 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:47.749698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21635
W20260927 16:37:47.755393 21620 connection.cc:535] client connection to 127.0.0.1:44053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44053): Connection reset by peer (error 104)
2026-09-27T16:37:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21734](info): set up 2 sockets
Sep 27 16:37:47 dist-test-slave-t27c krb5kdc[21734](info): commencing operation
krb5kdc: starting...
W20260927 16:37:48.794454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:48 dist-test-slave-t27c krb5kdc[21734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:48.920790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:49.034102 21747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:49.034492 21747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:49.034582 21747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:49.039299 21747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:49.039368 21747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:49.039389 21747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:49.039444 21747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:49.039482 21747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:49.045231 21747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:49.046550 21747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:49.047694 21747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:49.056553 21752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:49.056559 21755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:49.056672 21747 server_base.cc:1034] running on GCE node
W20260927 16:37:49.056603 21753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:49.057411 21747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:49.058594 21747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:49.059749 21747 hybrid_clock.cc:648] HybridClock initialized: now 1790527069059720 us; error 46 us; skew 500 ppm
Sep 27 16:37:49 dist-test-slave-t27c krb5kdc[21734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:49.062491 21747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:49.063331 21747 webserver.cc:460] Webserver started at http://127.0.0.1:36403/ using document root <none> and password file <none>
I20260927 16:37:49.063854 21747 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:49.063913 21747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:49.064078 21747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:49.066093 21747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "c76477e0ff0144b6a3e7e37c6be59f47"
format_stamp: "Formatted at 2026-09-27 16:37:49 on dist-test-slave-t27c"
server_key: "36184af18cd0cc83d1f6740a2dc39255"
server_key_iv: "b9a0f796a1e4bcb4420d9a411caad3b2"
server_key_version: "encryptionkey@0"
I20260927 16:37:49.066759 21747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "c76477e0ff0144b6a3e7e37c6be59f47"
format_stamp: "Formatted at 2026-09-27 16:37:49 on dist-test-slave-t27c"
server_key: "36184af18cd0cc83d1f6740a2dc39255"
server_key_iv: "b9a0f796a1e4bcb4420d9a411caad3b2"
server_key_version: "encryptionkey@0"
I20260927 16:37:49.070495 21747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:49.072907 21762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:49.074100 21747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:49.074281 21747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "c76477e0ff0144b6a3e7e37c6be59f47"
format_stamp: "Formatted at 2026-09-27 16:37:49 on dist-test-slave-t27c"
server_key: "36184af18cd0cc83d1f6740a2dc39255"
server_key_iv: "b9a0f796a1e4bcb4420d9a411caad3b2"
server_key_version: "encryptionkey@0"
I20260927 16:37:49.074390 21747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:49.093497 21747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:49.096516 21747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:49.096730 21747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:49.104417 21747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37803
I20260927 16:37:49.104439 21814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37803 every 8 connection(s)
I20260927 16:37:49.105433 21747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:49.107261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21747
I20260927 16:37:49.107410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:49.107748 392 external_mini_cluster.cc:1428] Setting key 1c3260dba6fae6a9fbdc5e2007e9b87f
I20260927 16:37:49.108551 21815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:49.115378 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: Bootstrap starting.
I20260927 16:37:49.117132 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:49.109161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:55089 (local address 127.0.0.1:37803)
0927 16:37:49.109611 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:49.109627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:49.109847 (+ 220us) server_negotiation.cc:407] Connection header received
0927 16:37:49.110654 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:49.110672 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:49.111020 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:49.111298 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:49.111636 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:49.112608 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:49.113795 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:49.114331 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:49.114448 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:49.115378 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:49.115384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:49.115396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:49.115841 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:49.115894 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:49.115916 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:49.116032 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:49.116259 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 16:37:49.116399 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260927 16:37:49.117667 21815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:49.118494 21815 log.cc:824] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:49.120182 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:49.120507 21815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: No bootstrap required, opened a new log
I20260927 16:37:49.125213 21815 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260927 16:37:49.125567 21815 raft_consensus.cc:374] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:49.125629 21815 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76477e0ff0144b6a3e7e37c6be59f47, State: Initialized, Role: FOLLOWER
I20260927 16:37:49.126151 21815 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260927 16:37:49.126358 21815 raft_consensus.cc:388] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:49.126441 21815 raft_consensus.cc:482] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:49.126543 21815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:49.126729 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:49.122364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:38323 (local address 127.0.0.1:37803)
0927 16:37:49.122494 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:49.122498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:49.122710 (+ 212us) server_negotiation.cc:407] Connection header received
0927 16:37:49.122794 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:49.122797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:49.122854 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:49.122966 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:49.123188 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:49.123976 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:49.124798 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:49.125124 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:49.125227 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:49.126219 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:49.126224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:49.126227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:49.126246 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:49.126271 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:49.126282 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:49.126394 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:49.126533 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 16:37:49.126574 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:49.127609 21815 raft_consensus.cc:504] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260927 16:37:49.128005 21815 leader_election.cc:302] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c76477e0ff0144b6a3e7e37c6be59f47; no voters:
I20260927 16:37:49.128355 21815 leader_election.cc:288] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:49.128477 21825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:49.128719 21825 raft_consensus.cc:686] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [term 1 LEADER]: Becoming Leader. State: Replica: c76477e0ff0144b6a3e7e37c6be59f47, State: Running, Role: LEADER
I20260927 16:37:49.129106 21825 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260927 16:37:49.129709 21815 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:49.130897 21827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76477e0ff0144b6a3e7e37c6be59f47. Latest consensus state: current_term: 1 leader_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } } }
I20260927 16:37:49.131059 21827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:49.130854 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76477e0ff0144b6a3e7e37c6be59f47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } } }
I20260927 16:37:49.131294 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47 [sys.catalog]: This master's current role is: LEADER
W20260927 16:37:49.134428 21840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:37:49.134496 21840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:37:49.134573 21838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:49.135272 21838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:49.141003 21838 catalog_manager.cc:1347] Generated new cluster ID: ff818ac3117b40c0bf44fc3c80b573e2
I20260927 16:37:49.141077 21838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:49.180037 21838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:49.181087 21838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:49.188026 21838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c76477e0ff0144b6a3e7e37c6be59f47: Generated new TSK 0
I20260927 16:37:49.188781 21838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:49.195384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21747
W20260927 16:37:49.200991 21732 connection.cc:535] client connection to 127.0.0.1:37803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37803): Connection reset by peer (error 104)
2026-09-27T16:37:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:49 dist-test-slave-t27c krb5kdc[21846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:49 dist-test-slave-t27c krb5kdc[21846](info): set up 2 sockets
Sep 27 16:37:49 dist-test-slave-t27c krb5kdc[21846](info): commencing operation
krb5kdc: starting...
W20260927 16:37:50.242496 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:50 dist-test-slave-t27c krb5kdc[21846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:37:50.369319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:50.483947 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:50.484301 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:50.484390 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:50.488569 21859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:50.488693 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:50.488734 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:50.488772 21859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:50.488806 21859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:50.494349 21859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:50.495491 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:50.496568 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:50.505468 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:50.505491 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:50.505477 21867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:50.505877 21859 server_base.cc:1034] running on GCE node
I20260927 16:37:50.506510 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:50.507685 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:50.508880 21859 hybrid_clock.cc:648] HybridClock initialized: now 1790527070508843 us; error 52 us; skew 500 ppm
Sep 27 16:37:50 dist-test-slave-t27c krb5kdc[21846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:50.518144 21859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:37:50.519245 21859 webserver.cc:460] Webserver started at http://127.0.0.1:45313/ using document root <none> and password file <none>
I20260927 16:37:50.519834 21859 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:50.519897 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:50.520073 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:50.522266 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "4f50925b745a4b44b582e0176b6fcb73"
format_stamp: "Formatted at 2026-09-27 16:37:50 on dist-test-slave-t27c"
server_key: "6bef92c43af1b4fe427ddf6910568108"
server_key_iv: "372f15c500c3082e2bdb22d785ed442d"
server_key_version: "encryptionkey@0"
I20260927 16:37:50.522835 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f50925b745a4b44b582e0176b6fcb73"
format_stamp: "Formatted at 2026-09-27 16:37:50 on dist-test-slave-t27c"
server_key: "6bef92c43af1b4fe427ddf6910568108"
server_key_iv: "372f15c500c3082e2bdb22d785ed442d"
server_key_version: "encryptionkey@0"
I20260927 16:37:50.526810 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 16:37:50.529328 21874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:50.530689 21859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:37:50.530795 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "4f50925b745a4b44b582e0176b6fcb73"
format_stamp: "Formatted at 2026-09-27 16:37:50 on dist-test-slave-t27c"
server_key: "6bef92c43af1b4fe427ddf6910568108"
server_key_iv: "372f15c500c3082e2bdb22d785ed442d"
server_key_version: "encryptionkey@0"
I20260927 16:37:50.530892 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:50.535442 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:50.538324 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:50.538476 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:50.545568 21859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37465
I20260927 16:37:50.545595 21926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37465 every 8 connection(s)
I20260927 16:37:50.546530 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:50.549122 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:50.554684 21927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: Bootstrap starting.
I20260927 16:37:50.555761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21859
I20260927 16:37:50.555893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:50.556182 392 external_mini_cluster.cc:1428] Setting key 41c5b8ee10db9ed46857f5433a7cab22
I20260927 16:37:50.557391 21927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:50.558303 21927 log.cc:824] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:50.560469 21927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: No bootstrap required, opened a new log
I20260927 16:37:50.561671 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:50.557300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:39913 (local address 127.0.0.1:37465)
0927 16:37:50.557804 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:50.557822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:50.557862 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:37:50.558769 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:50.558789 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:50.559104 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:50.559475 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:50.560170 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:50.560176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:50.560190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:50.560201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:50.560268 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:50.560295 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:50.560405 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:50.560486 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 16:37:50.560789 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":175,"threads_started":1}
I20260927 16:37:50.564463 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:50.564812 21927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } }
I20260927 16:37:50.565061 21927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:50.565140 21927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f50925b745a4b44b582e0176b6fcb73, State: Initialized, Role: FOLLOWER
I20260927 16:37:50.565577 21927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } }
I20260927 16:37:50.565723 21927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:50.565794 21927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:50.565923 21927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:50.567060 21927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } }
I20260927 16:37:50.567469 21927 leader_election.cc:302] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f50925b745a4b44b582e0176b6fcb73; no voters:
I20260927 16:37:50.567867 21927 leader_election.cc:288] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:50.568039 21937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:50.568254 21937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [term 1 LEADER]: Becoming Leader. State: Replica: 4f50925b745a4b44b582e0176b6fcb73, State: Running, Role: LEADER
I20260927 16:37:50.568639 21937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } }
I20260927 16:37:50.568751 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:50.566813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:56199 (local address 127.0.0.1:37465)
0927 16:37:50.566948 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:50.566953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:50.567140 (+ 187us) server_negotiation.cc:407] Connection header received
0927 16:37:50.567260 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:50.567263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:50.567307 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:50.567372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:50.568082 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:50.568085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:50.568088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:50.568103 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:50.568133 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:50.568147 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:50.568526 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:50.568559 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 16:37:50.568595 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:37:50.569283 21927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:50.569890 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f50925b745a4b44b582e0176b6fcb73. Latest consensus state: current_term: 1 leader_uuid: "4f50925b745a4b44b582e0176b6fcb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } } }
I20260927 16:37:50.570089 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:50.570340 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f50925b745a4b44b582e0176b6fcb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f50925b745a4b44b582e0176b6fcb73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37465 } } }
I20260927 16:37:50.570427 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73 [sys.catalog]: This master's current role is: LEADER
W20260927 16:37:50.574250 21952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:37:50.574311 21952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:37:50.574380 21946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:50.575024 21946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:50.580407 21946 catalog_manager.cc:1347] Generated new cluster ID: b8c2aa6c8f8a4696858fc993f1870c03
I20260927 16:37:50.580471 21946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:50.598129 21946 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:50.598443 21946 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:50.602720 21946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f50925b745a4b44b582e0176b6fcb73: Generated new TSK 0
I20260927 16:37:50.603345 21946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:50.616261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21859
2026-09-27T16:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:50 dist-test-slave-t27c krb5kdc[21958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:50 dist-test-slave-t27c krb5kdc[21958](info): set up 2 sockets
Sep 27 16:37:50 dist-test-slave-t27c krb5kdc[21958](info): commencing operation
krb5kdc: starting...
W20260927 16:37:51.670296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:51 dist-test-slave-t27c krb5kdc[21958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:51.789398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:51.899082 21971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:51.899363 21971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:51.899439 21971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:51.903455 21971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:51.903517 21971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:51.903535 21971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:51.903582 21971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:51.903613 21971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:51.908735 21971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:51.909765 21971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:51.910847 21971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:37:51.919034 21971 server_base.cc:1034] running on GCE node
W20260927 16:37:51.918983 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:51.918988 21979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:51.919003 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:51.919755 21971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:51.920902 21971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:51.922058 21971 hybrid_clock.cc:648] HybridClock initialized: now 1790527071922042 us; error 32 us; skew 500 ppm
Sep 27 16:37:51 dist-test-slave-t27c krb5kdc[21958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:51.924747 21971 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:51.925666 21971 webserver.cc:460] Webserver started at http://127.0.98.62:43799/ using document root <none> and password file <none>
I20260927 16:37:51.926302 21971 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:51.926368 21971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:51.926613 21971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:51.928690 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "ceea45e030284b1391301c6fee168315"
format_stamp: "Formatted at 2026-09-27 16:37:51 on dist-test-slave-t27c"
server_key: "c93482967ef89faee4f6f40f509a2f14"
server_key_iv: "229c7cc40b883e079e0f67570f42538b"
server_key_version: "encryptionkey@0"
I20260927 16:37:51.929265 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "ceea45e030284b1391301c6fee168315"
format_stamp: "Formatted at 2026-09-27 16:37:51 on dist-test-slave-t27c"
server_key: "c93482967ef89faee4f6f40f509a2f14"
server_key_iv: "229c7cc40b883e079e0f67570f42538b"
server_key_version: "encryptionkey@0"
I20260927 16:37:51.933071 21971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 16:37:51.935588 21986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:51.936764 21971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:51.936888 21971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "ceea45e030284b1391301c6fee168315"
format_stamp: "Formatted at 2026-09-27 16:37:51 on dist-test-slave-t27c"
server_key: "c93482967ef89faee4f6f40f509a2f14"
server_key_iv: "229c7cc40b883e079e0f67570f42538b"
server_key_version: "encryptionkey@0"
I20260927 16:37:51.936995 21971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:51.968626 21971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:51.971776 21971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:51.972000 21971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:51.979445 21971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39619
I20260927 16:37:51.979463 22038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39619 every 8 connection(s)
I20260927 16:37:51.980422 21971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:51.983139 22039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:51.985368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 21971
I20260927 16:37:51.985517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:51.985771 392 external_mini_cluster.cc:1428] Setting key e31ea8bc54d2b584cedcde257ab0053e
I20260927 16:37:51.989362 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315: Bootstrap starting.
I20260927 16:37:51.991935 22039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:51.992782 22039 log.cc:824] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:51.995019 22039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315: No bootstrap required, opened a new log
Sep 27 16:37:51 dist-test-slave-t27c krb5kdc[21958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:51.998633 22039 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } }
I20260927 16:37:51.998989 22039 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:51.999055 22039 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceea45e030284b1391301c6fee168315, State: Initialized, Role: FOLLOWER
I20260927 16:37:51.999599 22039 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } }
I20260927 16:37:51.999750 22039 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:51.999831 22039 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:51.999941 22039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:52.001135 22039 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } }
I20260927 16:37:52.001490 22039 leader_election.cc:302] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceea45e030284b1391301c6fee168315; no voters:
I20260927 16:37:52.001840 22039 leader_election.cc:288] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:52.001953 22044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:52.002228 22044 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [term 1 LEADER]: Becoming Leader. State: Replica: ceea45e030284b1391301c6fee168315, State: Running, Role: LEADER
I20260927 16:37:52.002236 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:51.987202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48439 (local address 127.0.98.62:39619)
0927 16:37:51.987632 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:51.987651 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:51.987694 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:37:51.988506 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:51.988528 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:51.988936 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:51.989287 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:51.990017 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:51.991529 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:51.992427 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:51.993121 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:51.993264 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:51.996274 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:51.996279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:51.996292 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:51.996330 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:51.998649 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:51.999261 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:51.999267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:51.999274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:51.999445 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:51.999821 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:51.999825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:51.999826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:52.000371 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:52.000596 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:52.000882 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 16:37:52.001121 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260927 16:37:52.002604 22044 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } }
I20260927 16:37:52.003275 22039 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:52.004375 22046 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceea45e030284b1391301c6fee168315. Latest consensus state: current_term: 1 leader_uuid: "ceea45e030284b1391301c6fee168315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } } }
I20260927 16:37:52.004753 22046 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:52.004657 22045 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceea45e030284b1391301c6fee168315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceea45e030284b1391301c6fee168315" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39619 } } }
I20260927 16:37:52.005118 22045 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:52.005817 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:52.006283 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:52.009016 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:52.015750 22055 catalog_manager.cc:1347] Generated new cluster ID: 9436751f06834addba75dbe47f04561f
I20260927 16:37:52.015817 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:52.017468 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:52.008375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53681 (local address 127.0.98.62:39619)
0927 16:37:52.008500 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:52.008506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:52.008739 (+ 233us) server_negotiation.cc:407] Connection header received
0927 16:37:52.008907 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:52.008910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:52.008961 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:52.009045 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:52.009734 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:52.010367 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:52.011302 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:52.011683 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:52.011801 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:52.013539 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:52.013542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:52.013544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:52.013570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:52.015266 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:52.015990 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:52.015993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:52.015995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:52.016089 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:52.016487 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:52.016490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:52.016492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:52.016744 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:52.016921 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:52.017165 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:37:52.017268 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260927 16:37:52.037953 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:52.038949 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:52.045286 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceea45e030284b1391301c6fee168315: Generated new TSK 0
I20260927 16:37:52.046315 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:52.054139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 21971
W20260927 16:37:52.059567 21956 connection.cc:535] client connection to 127.0.98.62:39619 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39619): Connection reset by peer (error 104)
2026-09-27T16:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:52 dist-test-slave-t27c krb5kdc[22070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:52 dist-test-slave-t27c krb5kdc[22070](info): set up 2 sockets
Sep 27 16:37:52 dist-test-slave-t27c krb5kdc[22070](info): commencing operation
krb5kdc: starting...
W20260927 16:37:53.102473 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:53 dist-test-slave-t27c krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:53.224879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:53.334741 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:53.335080 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:53.335165 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:53.339298 22083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:53.339383 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:53.339423 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:53.339486 22083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:53.339524 22083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:53.344866 22083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:53.345968 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:53.347033 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:53.355474 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:53.355501 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:53.355542 22083 server_base.cc:1034] running on GCE node
W20260927 16:37:53.355465 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:53.356356 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:53.357381 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:53.358542 22083 hybrid_clock.cc:648] HybridClock initialized: now 1790527073358511 us; error 43 us; skew 500 ppm
Sep 27 16:37:53 dist-test-slave-t27c krb5kdc[22070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:53.361222 22083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:53.362114 22083 webserver.cc:460] Webserver started at http://127.0.98.62:44073/ using document root <none> and password file <none>
I20260927 16:37:53.362720 22083 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:53.362812 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:53.363026 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:53.365087 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "6ffcc03b5fbb47b992896a90ba403c7e"
format_stamp: "Formatted at 2026-09-27 16:37:53 on dist-test-slave-t27c"
server_key: "cb77a77bbbac220a67b27f39c8cad60a"
server_key_iv: "86314d5779367f0299e4e4a6ebc7d6cd"
server_key_version: "encryptionkey@0"
I20260927 16:37:53.365676 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ffcc03b5fbb47b992896a90ba403c7e"
format_stamp: "Formatted at 2026-09-27 16:37:53 on dist-test-slave-t27c"
server_key: "cb77a77bbbac220a67b27f39c8cad60a"
server_key_iv: "86314d5779367f0299e4e4a6ebc7d6cd"
server_key_version: "encryptionkey@0"
I20260927 16:37:53.369643 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:37:53.372351 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:53.373682 22083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:53.373822 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "6ffcc03b5fbb47b992896a90ba403c7e"
format_stamp: "Formatted at 2026-09-27 16:37:53 on dist-test-slave-t27c"
server_key: "cb77a77bbbac220a67b27f39c8cad60a"
server_key_iv: "86314d5779367f0299e4e4a6ebc7d6cd"
server_key_version: "encryptionkey@0"
I20260927 16:37:53.373943 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:53.388237 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:53.391000 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:53.391199 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:53.398067 22083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35785
I20260927 16:37:53.398123 22150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35785 every 8 connection(s)
I20260927 16:37:53.399088 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:53.401006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22083
I20260927 16:37:53.401149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:53.401428 392 external_mini_cluster.cc:1428] Setting key e15d8d51918608204d985513e2e0fc20
I20260927 16:37:53.402242 22151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:53.408172 22151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e: Bootstrap starting.
I20260927 16:37:53.410604 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:53.402844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50653 (local address 127.0.98.62:35785)
0927 16:37:53.403290 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:53.403305 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:53.403341 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:37:53.403989 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:53.404005 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:53.404337 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:53.404675 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:53.405085 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:53.406084 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:53.407052 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:53.407555 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:53.407740 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:53.408749 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:53.408754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:53.408768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:53.409181 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:53.409238 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:53.409262 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:53.409410 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:53.409603 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 16:37:53.409755 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":149,"threads_started":1}
I20260927 16:37:53.410691 22151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:53.411595 22151 log.cc:824] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:53.413712 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:53.413800 22151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e: No bootstrap required, opened a new log
I20260927 16:37:53.418051 22151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260927 16:37:53.418365 22151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:53.418447 22151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffcc03b5fbb47b992896a90ba403c7e, State: Initialized, Role: FOLLOWER
I20260927 16:37:53.418938 22151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260927 16:37:53.419121 22151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:53.419212 22151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:53.419335 22151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:53.420030 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:53.415827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55643 (local address 127.0.98.62:35785)
0927 16:37:53.415943 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:53.415946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:53.416114 (+ 168us) server_negotiation.cc:407] Connection header received
0927 16:37:53.416248 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:53.416250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:53.416285 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:53.416348 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:53.416633 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:53.417117 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:53.418241 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:53.418540 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:53.418629 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:53.419576 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:53.419580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:53.419583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:53.419593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:53.419622 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:53.419636 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:53.419724 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:53.419868 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 16:37:53.419902 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:37:53.420468 22151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260927 16:37:53.420804 22151 leader_election.cc:302] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ffcc03b5fbb47b992896a90ba403c7e; no voters:
I20260927 16:37:53.421151 22151 leader_election.cc:288] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:53.421216 22161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:53.421391 22161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [term 1 LEADER]: Becoming Leader. State: Replica: 6ffcc03b5fbb47b992896a90ba403c7e, State: Running, Role: LEADER
I20260927 16:37:53.421684 22161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } }
I20260927 16:37:53.422456 22151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:53.423419 22162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ffcc03b5fbb47b992896a90ba403c7e. Latest consensus state: current_term: 1 leader_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260927 16:37:53.423542 22162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:53.423846 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffcc03b5fbb47b992896a90ba403c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35785 } } }
I20260927 16:37:53.423945 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:53.424028 22170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:53.427318 22170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:53.433051 22170 catalog_manager.cc:1347] Generated new cluster ID: aa2b883817a340a187e0a2dfe32c464d
I20260927 16:37:53.433128 22170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:53.439018 22170 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:53.440146 22170 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:53.444048 22170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ffcc03b5fbb47b992896a90ba403c7e: Generated new TSK 0
I20260927 16:37:53.444674 22170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:53.450088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22083
W20260927 16:37:53.454723 22068 connection.cc:535] client connection to 127.0.98.62:35785 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35785): Connection reset by peer (error 104)
2026-09-27T16:37:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:53 dist-test-slave-t27c krb5kdc[22182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:53 dist-test-slave-t27c krb5kdc[22182](info): set up 2 sockets
Sep 27 16:37:53 dist-test-slave-t27c krb5kdc[22182](info): commencing operation
krb5kdc: starting...
W20260927 16:37:54.490485 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:54 dist-test-slave-t27c krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:54.611017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:54.722913 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:54.723201 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:54.723286 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:54.727363 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:54.727437 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:54.727458 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:54.727474 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:54.727489 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:54.732999 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:54.734258 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:54.735391 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:54.743916 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:54.743921 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:54.743935 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:54.744107 22195 server_base.cc:1034] running on GCE node
I20260927 16:37:54.744756 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:54.745859 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:54.747051 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790527074747018 us; error 49 us; skew 500 ppm
Sep 27 16:37:54 dist-test-slave-t27c krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:54.749650 22195 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:54.750581 22195 webserver.cc:460] Webserver started at http://127.0.98.62:36089/ using document root <none> and password file <none>
I20260927 16:37:54.751142 22195 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:54.751232 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:54.751463 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:54.753505 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "b867e68a56034d9eac2e58418a86e795"
format_stamp: "Formatted at 2026-09-27 16:37:54 on dist-test-slave-t27c"
server_key: "0da672c52538740f6af675df10b516eb"
server_key_iv: "d596c00b1bbae2182b50660336da8eaa"
server_key_version: "encryptionkey@0"
I20260927 16:37:54.754087 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "b867e68a56034d9eac2e58418a86e795"
format_stamp: "Formatted at 2026-09-27 16:37:54 on dist-test-slave-t27c"
server_key: "0da672c52538740f6af675df10b516eb"
server_key_iv: "d596c00b1bbae2182b50660336da8eaa"
server_key_version: "encryptionkey@0"
I20260927 16:37:54.757663 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:37:54.760144 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:54.761329 22195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:37:54.761539 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "b867e68a56034d9eac2e58418a86e795"
format_stamp: "Formatted at 2026-09-27 16:37:54 on dist-test-slave-t27c"
server_key: "0da672c52538740f6af675df10b516eb"
server_key_iv: "d596c00b1bbae2182b50660336da8eaa"
server_key_version: "encryptionkey@0"
I20260927 16:37:54.761636 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:54.766100 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:54.769134 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:54.769349 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:54.776520 22195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44033
I20260927 16:37:54.776547 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44033 every 8 connection(s)
I20260927 16:37:54.777494 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:54.780328 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:54.785816 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795: Bootstrap starting.
I20260927 16:37:54.787220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22195
I20260927 16:37:54.787361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:54.787638 392 external_mini_cluster.cc:1428] Setting key 278c58ef0f125e2540dc5ff53a9f3cc1
I20260927 16:37:54.788700 22263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:54.789520 22263 log.cc:824] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:54.791580 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795: No bootstrap required, opened a new log
I20260927 16:37:54.792861 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:54.788768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40983 (local address 127.0.98.62:44033)
0927 16:37:54.789237 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:54.789252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:54.789288 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:37:54.790102 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:54.790128 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:54.790505 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:54.790847 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:54.791656 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:54.791661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:54.791693 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:54.791707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:54.791773 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:54.791794 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:54.791909 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:54.792016 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 16:37:54.792182 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260927 16:37:54.795681 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:54.795701 22263 raft_consensus.cc:348] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260927 16:37:54.795969 22263 raft_consensus.cc:374] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:54.796054 22263 raft_consensus.cc:729] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b867e68a56034d9eac2e58418a86e795, State: Initialized, Role: FOLLOWER
I20260927 16:37:54.796442 22263 consensus_queue.cc:260] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260927 16:37:54.796591 22263 raft_consensus.cc:388] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:54.796674 22263 raft_consensus.cc:482] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:54.796767 22263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:54.797713 22263 raft_consensus.cc:504] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260927 16:37:54.798079 22263 leader_election.cc:302] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b867e68a56034d9eac2e58418a86e795; no voters:
I20260927 16:37:54.798447 22263 leader_election.cc:288] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:54.798502 22273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:54.798745 22273 raft_consensus.cc:686] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [term 1 LEADER]: Becoming Leader. State: Replica: b867e68a56034d9eac2e58418a86e795, State: Running, Role: LEADER
I20260927 16:37:54.799419 22273 consensus_queue.cc:237] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260927 16:37:54.799621 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:54.797872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35105 (local address 127.0.98.62:44033)
0927 16:37:54.797984 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:54.797988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:54.798236 (+ 248us) server_negotiation.cc:407] Connection header received
0927 16:37:54.798407 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:54.798410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:54.798449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:54.798528 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:54.799233 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:54.799237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:54.799239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:54.799250 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:54.799275 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:54.799288 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:54.799388 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:54.799470 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 16:37:54.799505 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:37:54.799793 22263 sys_catalog.cc:564] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:54.801011 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b867e68a56034d9eac2e58418a86e795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260927 16:37:54.801139 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:54.801491 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b867e68a56034d9eac2e58418a86e795. Latest consensus state: current_term: 1 leader_uuid: "b867e68a56034d9eac2e58418a86e795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b867e68a56034d9eac2e58418a86e795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260927 16:37:54.801579 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:54.801908 22283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:54.805330 22283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:54.811625 22283 catalog_manager.cc:1347] Generated new cluster ID: fbd3bb13d8da4155860d943e588f9dae
I20260927 16:37:54.811714 22283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:54.840560 22283 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:54.840960 22283 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:54.852743 22283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b867e68a56034d9eac2e58418a86e795: Generated new TSK 0
I20260927 16:37:54.853667 22283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:54.864431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22195
2026-09-27T16:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:54 dist-test-slave-t27c krb5kdc[22294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:54 dist-test-slave-t27c krb5kdc[22294](info): set up 2 sockets
Sep 27 16:37:54 dist-test-slave-t27c krb5kdc[22294](info): commencing operation
krb5kdc: starting...
W20260927 16:37:55.930526 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:56.054127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:56.170935 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:56.171249 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:56.171291 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:56.175537 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:56.175608 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:56.175660 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:56.175712 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:56.175750 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:56.182418 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:56.183882 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:56.185016 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:56.193595 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:56.193609 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:56.193607 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:56.193897 22307 server_base.cc:1034] running on GCE node
I20260927 16:37:56.194480 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:56.195623 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:56.196806 22307 hybrid_clock.cc:648] HybridClock initialized: now 1790527076196797 us; error 33 us; skew 500 ppm
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:56.205268 22307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:56.206326 22307 webserver.cc:460] Webserver started at http://127.0.98.62:44011/ using document root <none> and password file <none>
I20260927 16:37:56.206894 22307 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:56.206979 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:56.207188 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:56.209283 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "34336c6162e74a8cafa91fec9adb9cc3"
format_stamp: "Formatted at 2026-09-27 16:37:56 on dist-test-slave-t27c"
server_key: "c698725936d8a08f7b391b7016c97249"
server_key_iv: "e8fe0e5c4056f92934249d39da3846c4"
server_key_version: "encryptionkey@0"
I20260927 16:37:56.209887 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "34336c6162e74a8cafa91fec9adb9cc3"
format_stamp: "Formatted at 2026-09-27 16:37:56 on dist-test-slave-t27c"
server_key: "c698725936d8a08f7b391b7016c97249"
server_key_iv: "e8fe0e5c4056f92934249d39da3846c4"
server_key_version: "encryptionkey@0"
I20260927 16:37:56.213967 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 16:37:56.216663 22322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:56.217965 22307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:56.218101 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "34336c6162e74a8cafa91fec9adb9cc3"
format_stamp: "Formatted at 2026-09-27 16:37:56 on dist-test-slave-t27c"
server_key: "c698725936d8a08f7b391b7016c97249"
server_key_iv: "e8fe0e5c4056f92934249d39da3846c4"
server_key_version: "encryptionkey@0"
I20260927 16:37:56.218261 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:56.235004 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:56.237762 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:56.237969 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:56.244817 22307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39537
I20260927 16:37:56.244843 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39537 every 8 connection(s)
I20260927 16:37:56.245748 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:56.248510 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:56.250550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22307
I20260927 16:37:56.250700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:56.250994 392 external_mini_cluster.cc:1428] Setting key ecb258731cf28aa55113315a3ce35863
I20260927 16:37:56.255406 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3: Bootstrap starting.
I20260927 16:37:56.257694 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:56.258658 22375 log.cc:824] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:37:56.260704 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3: No bootstrap required, opened a new log
I20260927 16:37:56.264418 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 16:37:56.264667 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:56.264737 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34336c6162e74a8cafa91fec9adb9cc3, State: Initialized, Role: FOLLOWER
I20260927 16:37:56.265333 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:56.252515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56797 (local address 127.0.98.62:39537)
0927 16:37:56.253002 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:56.253017 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:56.253056 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:37:56.253715 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:56.253735 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:56.254106 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:56.254509 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:56.255335 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:56.256356 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:56.257234 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:56.257714 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:56.257891 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:56.260650 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:56.260655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:56.260669 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:56.260694 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:56.262442 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:56.262879 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:56.262884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:56.262888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:56.262994 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:56.263258 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:56.263262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:56.263264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:56.263645 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:56.263855 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:56.264145 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 16:37:56.264418 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":145,"threads_started":1}
I20260927 16:37:56.265290 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 16:37:56.265482 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:56.265583 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:56.265694 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:56.266767 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 16:37:56.267135 22375 leader_election.cc:302] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34336c6162e74a8cafa91fec9adb9cc3; no voters:
I20260927 16:37:56.267488 22375 leader_election.cc:288] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:56.267567 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:56.267756 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 34336c6162e74a8cafa91fec9adb9cc3, State: Running, Role: LEADER
I20260927 16:37:56.268096 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } }
I20260927 16:37:56.268270 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:56.268761 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:56.269793 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34336c6162e74a8cafa91fec9adb9cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260927 16:37:56.269904 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:56.270224 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34336c6162e74a8cafa91fec9adb9cc3. Latest consensus state: current_term: 1 leader_uuid: "34336c6162e74a8cafa91fec9adb9cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34336c6162e74a8cafa91fec9adb9cc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39537 } } }
I20260927 16:37:56.270335 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:56.270445 22393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:56.273277 22393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:56.279448 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:56.270424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41251 (local address 127.0.98.62:39537)
0927 16:37:56.270547 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:56.270551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:56.271137 (+ 586us) server_negotiation.cc:407] Connection header received
0927 16:37:56.271270 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:56.271275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:56.271326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:56.271421 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:56.272294 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:56.273075 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:56.274239 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:56.274476 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:56.274596 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:56.276113 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:56.276117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:56.276120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:37:56.276145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:56.277839 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:56.278271 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:56.278276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:56.278278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:56.278358 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:37:56.278622 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:37:56.278626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:37:56.278628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:37:56.278800 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:56.278932 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:56.279160 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 16:37:56.279277 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:37:56.283840 22393 catalog_manager.cc:1347] Generated new cluster ID: 1fce0dc97dfe4f14b7f5420bb992ce8d
I20260927 16:37:56.283912 22393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:56.291369 22393 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:56.292699 22393 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:56.300891 22393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34336c6162e74a8cafa91fec9adb9cc3: Generated new TSK 0
I20260927 16:37:56.301751 22393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:56.307021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22307
2026-09-27T16:37:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22406](info): set up 2 sockets
Sep 27 16:37:56 dist-test-slave-t27c krb5kdc[22406](info): commencing operation
krb5kdc: starting...
W20260927 16:37:57.362462 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:57 dist-test-slave-t27c krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:57.483124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:57.592264 22419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:57.592533 22419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:57.592574 22419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:57.596513 22419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:57.596572 22419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:57.596591 22419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:57.596607 22419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:57.596622 22419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:57.601850 22419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:57.602883 22419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:57.603838 22419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:57.612360 22424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:57.612341 22427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:57.612401 22419 server_base.cc:1034] running on GCE node
W20260927 16:37:57.612341 22425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:57.613247 22419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:57.614431 22419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:57.615588 22419 hybrid_clock.cc:648] HybridClock initialized: now 1790527077615571 us; error 32 us; skew 500 ppm
Sep 27 16:37:57 dist-test-slave-t27c krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:57.618396 22419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:57.619318 22419 webserver.cc:460] Webserver started at http://127.0.98.62:34255/ using document root <none> and password file <none>
I20260927 16:37:57.619879 22419 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:57.619940 22419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:57.620172 22419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:57.622241 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "f3fe0ead6876432fa02ec9511f9cd24e"
format_stamp: "Formatted at 2026-09-27 16:37:57 on dist-test-slave-t27c"
server_key: "4e5702005785218d10b5bd12abda85e1"
server_key_iv: "b746eeb92c4774f32b7afac11aaf3084"
server_key_version: "encryptionkey@0"
I20260927 16:37:57.622809 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3fe0ead6876432fa02ec9511f9cd24e"
format_stamp: "Formatted at 2026-09-27 16:37:57 on dist-test-slave-t27c"
server_key: "4e5702005785218d10b5bd12abda85e1"
server_key_iv: "b746eeb92c4774f32b7afac11aaf3084"
server_key_version: "encryptionkey@0"
I20260927 16:37:57.626411 22419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:37:57.628697 22434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:57.629858 22419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:37:57.629976 22419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "f3fe0ead6876432fa02ec9511f9cd24e"
format_stamp: "Formatted at 2026-09-27 16:37:57 on dist-test-slave-t27c"
server_key: "4e5702005785218d10b5bd12abda85e1"
server_key_iv: "b746eeb92c4774f32b7afac11aaf3084"
server_key_version: "encryptionkey@0"
I20260927 16:37:57.630071 22419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:57.657668 22419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:57.660784 22419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:57.660988 22419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:57.668323 22419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41643
I20260927 16:37:57.668349 22486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41643 every 8 connection(s)
I20260927 16:37:57.669283 22419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:57.672003 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:57.677281 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e: Bootstrap starting.
I20260927 16:37:57.679340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22419
I20260927 16:37:57.679476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:57.679764 392 external_mini_cluster.cc:1428] Setting key 647d282a7daf0ba73a9f973881f0afcb
I20260927 16:37:57.679766 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:57.680620 22487 log.cc:824] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:57.682585 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e: No bootstrap required, opened a new log
I20260927 16:37:57.686789 22487 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260927 16:37:57.687019 22487 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:57.687085 22487 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fe0ead6876432fa02ec9511f9cd24e, State: Initialized, Role: FOLLOWER
I20260927 16:37:57.687556 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260927 16:37:57.687711 22487 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:57.687819 22487 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:57.687932 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:57.688642 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:57.680833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36177 (local address 127.0.98.62:41643)
0927 16:37:57.681259 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:57.681275 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:57.681310 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:37:57.681943 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:57.681960 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:57.682371 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:57.682673 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:57.683023 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:57.683946 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:57.685220 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:57.685703 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:57.685834 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:57.686712 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:57.686718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:57.686731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:57.687226 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:57.687284 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:57.687307 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:57.687418 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:57.687579 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 16:37:57.687803 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260927 16:37:57.688905 22487 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260927 16:37:57.689263 22487 leader_election.cc:302] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3fe0ead6876432fa02ec9511f9cd24e; no voters:
I20260927 16:37:57.689580 22487 leader_election.cc:288] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:57.689703 22492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:57.689904 22492 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [term 1 LEADER]: Becoming Leader. State: Replica: f3fe0ead6876432fa02ec9511f9cd24e, State: Running, Role: LEADER
I20260927 16:37:57.690294 22492 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260927 16:37:57.691064 22487 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:57.691968 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:57.692206 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } } }
I20260927 16:37:57.692315 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:57.692644 22494 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3fe0ead6876432fa02ec9511f9cd24e. Latest consensus state: current_term: 1 leader_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fe0ead6876432fa02ec9511f9cd24e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } } }
I20260927 16:37:57.692725 22494 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:57.693547 22501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:57.696929 22501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:57.699021 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:57.694330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36299 (local address 127.0.98.62:41643)
0927 16:37:57.694450 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:57.694459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:57.694754 (+ 295us) server_negotiation.cc:407] Connection header received
0927 16:37:57.694889 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:57.694891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:57.694931 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:57.694998 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:57.695322 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:57.696124 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:57.697208 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:37:57.697549 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:37:57.697658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:37:57.698578 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:57.698582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:57.698585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:57.698598 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:57.698628 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:57.698641 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:57.698720 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:57.698830 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 16:37:57.698885 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:37:57.702662 22501 catalog_manager.cc:1347] Generated new cluster ID: 9803e812cc474328bea1e1f72e03b6f9
I20260927 16:37:57.702723 22501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:57.722772 22501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:57.724112 22501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:57.735662 22501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3fe0ead6876432fa02ec9511f9cd24e: Generated new TSK 0
I20260927 16:37:57.736606 22501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:57.743219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22419
2026-09-27T16:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:57 dist-test-slave-t27c krb5kdc[22518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:57 dist-test-slave-t27c krb5kdc[22518](info): set up 2 sockets
Sep 27 16:37:57 dist-test-slave-t27c krb5kdc[22518](info): commencing operation
krb5kdc: starting...
W20260927 16:37:58.798583 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:37:58 dist-test-slave-t27c krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:37:58.918995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:37:59.029966 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:37:59.030295 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:37:59.030364 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:37:59.034395 22531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:37:59.034461 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:37:59.034500 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:37:59.034533 22531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:37:59.034559 22531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:37:59.039842 22531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:37:59.040874 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:37:59.041915 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:37:59.050276 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:59.050302 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:37:59.050279 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:37:59.050354 22531 server_base.cc:1034] running on GCE node
I20260927 16:37:59.051100 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:37:59.052290 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:37:59.053489 22531 hybrid_clock.cc:648] HybridClock initialized: now 1790527079053469 us; error 38 us; skew 500 ppm
Sep 27 16:37:59 dist-test-slave-t27c krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:37:59.056449 22531 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:37:59.057371 22531 webserver.cc:460] Webserver started at http://127.0.98.62:40849/ using document root <none> and password file <none>
I20260927 16:37:59.057916 22531 fs_manager.cc:359] Metadata directory not provided
I20260927 16:37:59.057976 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:37:59.058148 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:37:59.060273 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "854574379e34403b976c2de7c2cacf9d"
format_stamp: "Formatted at 2026-09-27 16:37:59 on dist-test-slave-t27c"
server_key: "dfa8239893417e572603f3cba070cfbe"
server_key_iv: "8ac048d14689f91062f1383a87164e2b"
server_key_version: "encryptionkey@0"
I20260927 16:37:59.060796 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "854574379e34403b976c2de7c2cacf9d"
format_stamp: "Formatted at 2026-09-27 16:37:59 on dist-test-slave-t27c"
server_key: "dfa8239893417e572603f3cba070cfbe"
server_key_iv: "8ac048d14689f91062f1383a87164e2b"
server_key_version: "encryptionkey@0"
I20260927 16:37:59.064745 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:37:59.067435 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:37:59.068673 22531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:37:59.068848 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "854574379e34403b976c2de7c2cacf9d"
format_stamp: "Formatted at 2026-09-27 16:37:59 on dist-test-slave-t27c"
server_key: "dfa8239893417e572603f3cba070cfbe"
server_key_iv: "8ac048d14689f91062f1383a87164e2b"
server_key_version: "encryptionkey@0"
I20260927 16:37:59.068948 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:37:59.074728 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:37:59.078107 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:37:59.078344 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:37:59.085253 22531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41397
I20260927 16:37:59.085261 22598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41397 every 8 connection(s)
I20260927 16:37:59.086265 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:37:59.088986 22599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:37:59.094791 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d: Bootstrap starting.
I20260927 16:37:59.095060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22531
I20260927 16:37:59.095201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:37:59.095492 392 external_mini_cluster.cc:1428] Setting key f58209b2b96b547d0c29d9e18a5ae594
I20260927 16:37:59.097657 22599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d: Neither blocks nor log segments found. Creating new log.
I20260927 16:37:59.098466 22599 log.cc:824] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:37:59.100724 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d: No bootstrap required, opened a new log
I20260927 16:37:59.101101 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:59.096877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54321 (local address 127.0.98.62:41397)
0927 16:37:59.097316 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:59.097331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:59.097367 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:37:59.098108 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:59.098129 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:59.098502 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:59.098903 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:59.099811 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:59.099817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:59.099838 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:59.099848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:59.099900 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:59.099920 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:59.100042 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:59.100148 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:37:59.100310 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260927 16:37:59.103995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:37:59.104691 22599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260927 16:37:59.104919 22599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:37:59.104979 22599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854574379e34403b976c2de7c2cacf9d, State: Initialized, Role: FOLLOWER
I20260927 16:37:59.105458 22599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260927 16:37:59.105592 22599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:37:59.105672 22599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:37:59.105778 22599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:37:59.107084 22599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260927 16:37:59.107455 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 854574379e34403b976c2de7c2cacf9d; no voters:
I20260927 16:37:59.107831 22599 leader_election.cc:288] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:37:59.107879 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:37:59.106204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52459 (local address 127.0.98.62:41397)
0927 16:37:59.106321 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:37:59.106326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:37:59.106514 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:37:59.106631 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:37:59.106633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:37:59.106664 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:37:59.106728 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:37:59.107446 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:37:59.107449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:37:59.107458 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:37:59.107473 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:37:59.107502 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:37:59.107515 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:37:59.107607 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:37:59.107713 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:37:59.107754 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:37:59.108222 22610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:37:59.108412 22610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [term 1 LEADER]: Becoming Leader. State: Replica: 854574379e34403b976c2de7c2cacf9d, State: Running, Role: LEADER
I20260927 16:37:59.108781 22610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260927 16:37:59.109190 22599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:37:59.109934 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 854574379e34403b976c2de7c2cacf9d. Latest consensus state: current_term: 1 leader_uuid: "854574379e34403b976c2de7c2cacf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260927 16:37:59.110121 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:59.110450 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "854574379e34403b976c2de7c2cacf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854574379e34403b976c2de7c2cacf9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260927 16:37:59.110615 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d [sys.catalog]: This master's current role is: LEADER
I20260927 16:37:59.110563 22618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:37:59.113624 22618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:37:59.118758 22618 catalog_manager.cc:1347] Generated new cluster ID: 0d57cbf013fa446a9240bcef73eafd0c
I20260927 16:37:59.118829 22618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:37:59.129961 22618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:37:59.130348 22618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:37:59.141074 22618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 854574379e34403b976c2de7c2cacf9d: Generated new TSK 0
I20260927 16:37:59.142000 22618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:37:59.156188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22531
2026-09-27T16:37:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1413 ms)
[----------] 18 tests from AuthTokenIssuingTest (25604 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:37:59 dist-test-slave-t27c krb5kdc[22630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:37:59 dist-test-slave-t27c krb5kdc[22630](info): set up 2 sockets
Sep 27 16:37:59 dist-test-slave-t27c krb5kdc[22630](info): commencing operation
krb5kdc: starting...
W20260927 16:38:00.210642 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:00.331532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:00.447603 22643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:00.447923 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:00.447950 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:00.448052 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:00.452200 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:00.452272 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:00.452294 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:00.452312 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:00.452363 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:00.458019 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:00.459336 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:00.460490 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:00.469219 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:00.469237 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:00.469280 22643 server_base.cc:1034] running on GCE node
W20260927 16:38:00.469219 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:00.470086 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:00.471243 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:00.472447 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790527080472428 us; error 28 us; skew 500 ppm
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:00.475397 22643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:00.476281 22643 webserver.cc:460] Webserver started at http://127.0.98.62:38267/ using document root <none> and password file <none>
I20260927 16:38:00.476867 22643 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:00.476958 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:00.477200 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:00.479312 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "36ef2903f23e4b8598de35741f53018a"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "a1416949f11da436524bc0c7e2106983"
server_key_iv: "b2840095b06bd38cc25043141fbb904f"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.479876 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "36ef2903f23e4b8598de35741f53018a"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "a1416949f11da436524bc0c7e2106983"
server_key_iv: "b2840095b06bd38cc25043141fbb904f"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.483590 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:00.486065 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:00.487392 22643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:00.487533 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "36ef2903f23e4b8598de35741f53018a"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "a1416949f11da436524bc0c7e2106983"
server_key_iv: "b2840095b06bd38cc25043141fbb904f"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.487648 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:00.521678 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:00.524770 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:00.525085 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:00.532243 22643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42787
I20260927 16:38:00.532578 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42787 every 8 connection(s)
I20260927 16:38:00.533309 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:00.536245 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:00.538650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22643
I20260927 16:38:00.538790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:00.539093 392 external_mini_cluster.cc:1428] Setting key 8b6b4363db378e1c7861eaedc83a43a9
I20260927 16:38:00.541072 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:00.557278 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.543431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.0.1:35800)
0927 16:38:00.544009 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.544079 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.545357 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.546110 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.546115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.546632 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.547030 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.547053 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.548253 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.548255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.549049 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.549058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.549829 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.549836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.549933 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.550610 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:00.550626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:00.552238 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:00.554203 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.554213 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.554225 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.554445 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.554759 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.554762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.554764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.554874 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.555464 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:00.555470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:00.555617 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 16:38:00.555777 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 16:38:00.556213 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":151,"thread_start_us":112,"threads_started":1}
I20260927 16:38:00.557278 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.544164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.0.98.62:42787)
0927 16:38:00.544421 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.544436 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.544473 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:38:00.545698 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.545703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.545899 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.546053 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.547198 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.548131 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.549201 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.549651 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.549845 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.552387 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:00.552394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:00.552405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:00.552424 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:00.554092 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.554555 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.554565 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.554567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.554663 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.554966 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.554969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.554971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.555344 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:00.555491 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:00.555848 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 16:38:00.556166 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260927 16:38:00.559823 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:00.560582 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.560333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51312)
0927 16:38:00.560445 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.560519 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:38:00.560839 22661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
I20260927 16:38:00.561666 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.561476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51328)
0927 16:38:00.561568 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.561612 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 16:38:00.561815 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45329: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:38:00.565037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45329
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:38:00.611070 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } attempt: 1
I20260927 16:38:00.611992 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.611683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51336)
0927 16:38:00.611842 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.611913 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 16:38:00.612774 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.612547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51350)
0927 16:38:00.612658 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.612715 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:38:00.612943 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45329: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
W20260927 16:38:00.678155 22718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:00.678508 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:00.678532 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:00.678579 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:38:00.681121 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } attempt: 2
I20260927 16:38:00.681916 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.681624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51356)
0927 16:38:00.681775 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.681842 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 16:38:00.682684 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.682483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51364)
0927 16:38:00.682579 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.682626 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:38:00.682847 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45329: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45329: connect: Connection refused (error 111)
W20260927 16:38:00.683007 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:00.683061 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:00.683099 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:00.683130 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:00.683164 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:00.688844 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:00.689998 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:00.691167 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:00.700309 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:00.700290 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:00.700426 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:00.700558 22718 server_base.cc:1034] running on GCE node
I20260927 16:38:00.701131 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:00.702342 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:00.703521 22718 hybrid_clock.cc:648] HybridClock initialized: now 1790527080703485 us; error 49 us; skew 500 ppm
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:00.706221 22718 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:38:00.707103 22718 webserver.cc:460] Webserver started at http://127.0.98.61:35459/ using document root <none> and password file <none>
I20260927 16:38:00.707657 22718 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:00.707718 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:00.707886 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:00.709900 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data/instance:
uuid: "bc8a1701a2714e19a226e79ac9cc1441"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "479fbdd8e5316b3d959915635c657855"
server_key_iv: "86b7596e946249e9595c3eb05504f5dd"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.710487 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal/instance:
uuid: "bc8a1701a2714e19a226e79ac9cc1441"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "479fbdd8e5316b3d959915635c657855"
server_key_iv: "86b7596e946249e9595c3eb05504f5dd"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.714139 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:38:00.716604 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:00.717793 22718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:00.717921 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "bc8a1701a2714e19a226e79ac9cc1441"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "479fbdd8e5316b3d959915635c657855"
server_key_iv: "86b7596e946249e9595c3eb05504f5dd"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.718031 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:00.733963 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:00.737025 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:00.737241 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:00.744349 22718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45329
I20260927 16:38:00.744629 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45329 every 8 connection(s)
I20260927 16:38:00.745373 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:38:00.748159 22786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:00.751391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22718
I20260927 16:38:00.751487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-1/wal/instance
I20260927 16:38:00.751735 392 external_mini_cluster.cc:1428] Setting key 6db597f2cf1b4117bfb33f49764f527f
I20260927 16:38:00.751955 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:00.766987 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.753828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.0.98.62:42787)
0927 16:38:00.753984 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.753988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.754892 (+ 904us) server_negotiation.cc:407] Connection header received
0927 16:38:00.755816 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.755820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.755879 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.755980 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.757122 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.757648 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.758817 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.759055 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.759161 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.763319 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:00.763323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:00.763325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:00.763357 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:00.764859 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.765396 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.765399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.765400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.765473 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.765818 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.765822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.765823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.766018 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:00.766123 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:00.766685 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 16:38:00.766812 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:38:00.767814 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.754286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.0.1:35808)
0927 16:38:00.754739 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.754783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.755608 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.756148 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.756154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.756497 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.756969 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.756991 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.757771 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.757779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.758691 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.758701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.759208 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.759212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.759386 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.760000 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:00.760024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:00.763126 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:00.764994 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.765000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.765017 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.765277 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.765563 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.765567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.765569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.765722 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.766120 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:00.766126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:00.766429 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 16:38:00.766609 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:38:00.766954 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260927 16:38:00.769205 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:00.774021 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } attempt: 3
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 16:38:00.778047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43653
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:38:00.779366 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.769772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51380)
0927 16:38:00.769921 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.769931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.769989 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.770588 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.770590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.770721 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.770807 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.770811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.771817 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.771819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.772402 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.772410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.773062 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.773064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.773136 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.773607 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:00.773618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:00.775270 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:00.777594 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.777601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.777603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.777794 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.778080 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.778082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.778084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.778209 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.778814 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:00.778817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:00.778899 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 16:38:00.779023 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 16:38:00.779164 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:38:00.780457 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.769903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.98.61:45329)
0927 16:38:00.770252 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.770262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.770281 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:38:00.770314 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.770318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.770474 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.770582 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.770888 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.771720 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.772507 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.772959 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.773050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.775443 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:00.775467 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:00.775503 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:00.775530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:00.777489 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.777903 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.777909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.777911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.777985 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.778309 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.778311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.778313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.778618 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:00.778742 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:00.779972 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0927 16:38:00.780095 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":133,"threads_started":1}
I20260927 16:38:00.782440 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:00.783246 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.782922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40372)
0927 16:38:00.783069 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.783161 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 16:38:00.783569 22735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
I20260927 16:38:00.784361 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.774580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51384)
0927 16:38:00.774719 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.774737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.774833 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.775503 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.775506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.775711 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.775860 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.775868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.776933 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.776937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.777777 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.777791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.778467 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.778471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.778597 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.779072 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:00.779086 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:00.780654 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:00.782668 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.782672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.782674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.782934 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.783227 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:00.783232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:00.783234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:00.783344 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:00.783831 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:00.783837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:00.783943 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 16:38:00.784054 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 16:38:00.784201 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 16:38:00.784538 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.784264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40384)
0927 16:38:00.784391 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.784461 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 16:38:00.784755 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
I20260927 16:38:00.784958 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.774665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.0.98.61:45329)
0927 16:38:00.775172 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.775182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.775197 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:38:00.775284 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.775286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.775336 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.775470 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.775979 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.776788 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.777924 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.778319 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.778458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.780833 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:00.780840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:00.780845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:00.780876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:00.782522 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.783054 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.783057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.783059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.783138 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:00.783425 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:00.783431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:00.783433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:00.783646 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:00.783733 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:00.784625 (+ 892us) server_negotiation.cc:299] Negotiation successful
0927 16:38:00.784743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":294,"threads_started":1}
I20260927 16:38:00.785573 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:00.786299 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.786030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40390)
0927 16:38:00.786141 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.786234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 16:38:00.787274 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.787045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40392)
0927 16:38:00.787185 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.787226 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 16:38:00.787464 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
I20260927 16:38:00.818629 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 1
I20260927 16:38:00.819571 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.819240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40400)
0927 16:38:00.819403 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.819472 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 16:38:00.820322 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.820097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40406)
0927 16:38:00.820208 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.820260 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:38:00.820454 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
I20260927 16:38:00.834052 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 1
I20260927 16:38:00.834988 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.834661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40412)
0927 16:38:00.834822 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.834895 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 16:38:00.835662 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.835463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40428)
0927 16:38:00.835559 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.835607 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:38:00.835780 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
W20260927 16:38:00.893432 22793 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:00.893734 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:00.893759 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:00.893797 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:38:00.895623 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 2
I20260927 16:38:00.896481 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.896172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40436)
0927 16:38:00.896330 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.896394 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 16:38:00.897257 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.897043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40446)
0927 16:38:00.897149 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.897201 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 16:38:00.897406 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
W20260927 16:38:00.898391 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:00.898465 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:00.898523 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:00.898576 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:00.898613 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:00.903929 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 2
I20260927 16:38:00.904280 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42787,127.0.98.61:45329,127.0.98.60:43653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:00.904791 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.904408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40462)
0927 16:38:00.904647 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.904711 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 16:38:00.905434 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.905235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40476)
0927 16:38:00.905330 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.905379 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:38:00.905563 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43653: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43653: connect: Connection refused (error 111)
I20260927 16:38:00.905659 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:00.906665 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:00.914902 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:00.914906 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:00.914978 22793 server_base.cc:1034] running on GCE node
W20260927 16:38:00.915037 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:00.915669 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:00.916832 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:00.918016 22793 hybrid_clock.cc:648] HybridClock initialized: now 1790527080917983 us; error 46 us; skew 500 ppm
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:00.920719 22793 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:38:00.921629 22793 webserver.cc:460] Webserver started at http://127.0.98.60:42675/ using document root <none> and password file <none>
I20260927 16:38:00.922307 22793 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:00.922395 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:00.922622 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:00.924696 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data/instance:
uuid: "ed0c2bbd155948069eda58c553946de5"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "996866fa5e1f77fac8e1b50e25ef4f20"
server_key_iv: "b6f095cbc53de2d5120f1718ade6cb15"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.925269 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal/instance:
uuid: "ed0c2bbd155948069eda58c553946de5"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "996866fa5e1f77fac8e1b50e25ef4f20"
server_key_iv: "b6f095cbc53de2d5120f1718ade6cb15"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.928952 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:38:00.931401 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:00.932622 22793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:00.932781 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "ed0c2bbd155948069eda58c553946de5"
format_stamp: "Formatted at 2026-09-27 16:38:00 on dist-test-slave-t27c"
server_key: "996866fa5e1f77fac8e1b50e25ef4f20"
server_key_iv: "b6f095cbc53de2d5120f1718ade6cb15"
server_key_version: "encryptionkey@0"
I20260927 16:38:00.932904 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:00.971195 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:00.974512 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:00.974740 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:00.982578 22793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43653
I20260927 16:38:00.982859 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43653 every 8 connection(s)
I20260927 16:38:00.983691 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:38:00.985942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 22793
I20260927 16:38:00.986040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526802550915-392-0/minicluster-data/master-2/wal/instance
I20260927 16:38:00.986361 392 external_mini_cluster.cc:1428] Setting key b3424cd074355dd0e2cb9f240fc5650a
I20260927 16:38:00.986697 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:00.991230 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:38:00.996595 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 3
I20260927 16:38:00.996716 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } attempt: 3
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:38:00 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:38:01.003860 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.987779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.0.98.60:43653)
0927 16:38:00.988199 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.988213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.988249 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:38:00.988914 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.988931 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.989267 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.989600 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.990436 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.991664 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.992558 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.993135 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.993311 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.996253 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:00.996260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:00.996272 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:00.996303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:00.999709 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.000250 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.000256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.000262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.000383 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.000702 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.000708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.000712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.001302 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.002245 (+ 943us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.002413 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.002771 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":286,"thread_start_us":139,"threads_started":1}
I20260927 16:38:01.004470 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.992909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.0.1:35814)
0927 16:38:00.993327 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.993373 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.993529 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.993890 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.993897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.994443 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.994689 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.994701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.995413 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.995418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.996161 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.996171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.996895 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.996899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:00.997042 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:00.997658 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:00.997675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:01.000024 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:01.002607 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.002615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.002621 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.002908 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.003184 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.003187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.003189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.003275 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.003789 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:01.003795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:01.003924 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 16:38:01.004072 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 16:38:01.004244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":91,"threads_started":1}
I20260927 16:38:01.006692 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.992960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.0.98.62:42787)
0927 16:38:00.993094 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.993099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.993445 (+ 346us) server_negotiation.cc:407] Connection header received
0927 16:38:00.993659 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.993663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.993717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.993803 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.994820 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.995288 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.996266 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.996655 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.996771 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.000165 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.000168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.000171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.000195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.002517 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.003000 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.003006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.003008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.003088 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.003371 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.003374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.003376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.003673 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.003782 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.006384 (+ 2602us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.006518 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:38:01.007750 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:01.008406 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.997005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40500)
0927 16:38:00.997124 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.997139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.997214 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.998461 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.998464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.998666 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.998775 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.998782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.999839 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.999843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.000629 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.000640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.001293 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.001297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.001401 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.001942 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:01.001958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:01.003729 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:01.006552 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.006555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.006557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.006793 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.007359 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.007370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.007461 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.008024 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:01.008027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:01.008128 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 16:38:01.008209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:38:01.008307 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 16:38:01.008694 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.997166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40504)
0927 16:38:00.997678 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:00.997695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:00.997773 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:00.998116 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:00.998120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:00.998345 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:00.998464 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:00.998472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.999511 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:00.999514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.000411 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.000423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.001227 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.001233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.001332 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.001770 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:01.001786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:01.004173 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:01.006514 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.006517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.006520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.006764 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.007413 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.007418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.007510 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.008160 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:01.008163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:01.008278 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 16:38:01.008387 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 16:38:01.008541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417}
I20260927 16:38:01.009460 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.997503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.0.98.60:43653)
0927 16:38:00.997823 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.997832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.997845 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:38:00.997885 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.997889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.997928 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.998076 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.998562 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.999365 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.000554 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.001083 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.001555 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.004289 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.004292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.004295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.004320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.006379 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007203 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.007206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.007208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.007308 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007618 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.007620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.007622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.008045 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.009088 (+ 1043us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.009160 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.009266 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260927 16:38:01.009680 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:00.997207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.98.60:43653)
0927 16:38:00.998261 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:00.998271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:00.998281 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:38:00.998311 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:00.998312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:00.998340 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:00.998437 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:00.998909 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:00.999708 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.000759 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.001158 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.001612 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.004382 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.004385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.004388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.004412 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.006418 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007149 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.007152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.007153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.007248 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.007631 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.007636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.007640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.007856 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.009302 (+ 1446us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.009372 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.009496 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":1920,"thread_start_us":136,"threads_started":1}
I20260927 16:38:01.012749 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: Bootstrap starting.
I20260927 16:38:01.013216 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: Bootstrap starting.
Sep 27 16:38:01 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:38:01 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:38:01.015712 22711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:01.015712 22786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:01.016678 22711 log.cc:824] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:01.016678 22786 log.cc:824] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:01.018805 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.008727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45329 (local address 127.0.0.1:51388)
0927 16:38:01.009081 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:01.009095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:01.009173 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:01.009491 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:01.009494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:01.009641 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:01.009755 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.009764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.010886 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.010889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.011704 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.011719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.012508 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.012513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.012610 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.013042 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:01.013063 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:01.014576 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:01.017078 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.017084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.017325 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.017645 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.017655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.017746 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.018260 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:01.018263 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:01.018357 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:38:01.018457 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 16:38:01.018602 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260927 16:38:01.018934 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: No bootstrap required, opened a new log
I20260927 16:38:01.018934 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: No bootstrap required, opened a new log
I20260927 16:38:01.019383 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.008977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.0.98.61:45329)
0927 16:38:01.009098 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.009103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.009136 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:38:01.009271 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.009276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.009322 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.009388 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.010209 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.010768 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.011847 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.012266 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.012379 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.014741 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.014746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.014750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.014775 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.016954 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017437 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.017440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.017442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.017539 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017846 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.017849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.017851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.018098 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.018217 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.019077 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.019216 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:38:01.019747 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.007717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.0.98.62:42787)
0927 16:38:01.007823 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.007827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.007840 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:38:01.007954 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.007960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.008025 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.008093 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.008885 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.009602 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.010698 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.010998 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.011090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.014784 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.014788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.014791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.014816 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.016657 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017289 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.017293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.017295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.017379 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.017678 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.017682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.017687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.017915 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.018110 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.019310 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.019644 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:38:01.022862 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:01.023195 22711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.023582 22711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2903f23e4b8598de35741f53018a, State: Initialized, Role: FOLLOWER
I20260927 16:38:01.023955 22711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.024438 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.024600 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:01.024559 22786 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.024766 22786 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8a1701a2714e19a226e79ac9cc1441, State: Initialized, Role: FOLLOWER
I20260927 16:38:01.025070 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.025425 22711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:01.025558 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.025686 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 16:38:01.026813 22664 tablet.cc:2369] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:38:01.026844 22786 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:38:01.030717 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:01.030810 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 16:38:01 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:38:01 dist-test-slave-t27c krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260927 16:38:01.034137 22895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:01.034848 22895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:01.038064 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.023632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.61:45329)
0927 16:38:01.023781 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.023785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.026447 (+ 2662us) server_negotiation.cc:407] Connection header received
0927 16:38:01.026612 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.026619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.026670 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.026750 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.027738 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.028492 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.029581 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.029891 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.029965 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.033606 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.033610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.033612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.033638 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.035989 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.036475 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.036478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.036480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.036570 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.036893 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.036896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.036897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.037167 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.037379 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.037778 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.037898 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 16:38:01.038496 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.023373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43653 (local address 127.0.0.1:40512)
0927 16:38:01.023473 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:01.023488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:01.023582 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:01.027991 (+ 4409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:01.027998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:01.028211 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:01.028337 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.028346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.029678 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.029682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.030425 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:01.030436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.030979 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.030982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:01.031053 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.032508 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:01.032521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:01.033974 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:01.036002 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.036005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.036007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.036238 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.037555 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:01.037558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:01.037559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:01.037642 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:01.038029 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:01.038032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:01.038105 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 16:38:01.038210 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 16:38:01.038339 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 16:38:01.039142 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.023410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.0.98.60:43653)
0927 16:38:01.027740 (+ 4330us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.027746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.027766 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:38:01.027817 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.027820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.027866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.028446 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.028528 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.029292 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.030547 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.030859 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.031253 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.034096 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.034099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.034102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.034122 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.035871 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.037369 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.037372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.037373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.037460 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.037739 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.037741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.037743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.037937 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.038822 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.038893 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.039000 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4218}
I20260927 16:38:01.037521 22718 master_runner.cc:186] Error getting master registration for 127.0.98.60:43653: OK, instance_id { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" instance_seqno: 1790527080980532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:38:01.040720 22718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:38:01.042135 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: Bootstrap starting.
I20260927 16:38:01.040791 22643 master_runner.cc:186] Error getting master registration for 127.0.98.60:43653: OK, instance_id { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" instance_seqno: 1790527080980532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:38:01.043283 22643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:38:01.045013 22862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:01.045912 22862 log.cc:824] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:01.046424 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:38:01.047628 22685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "36ef2903f23e4b8598de35741f53018a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35830
I20260927 16:38:01.047881 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:38:01.048017 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:01.049392 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.051051 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36ef2903f23e4b8598de35741f53018a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc8a1701a2714e19a226e79ac9cc1441"
I20260927 16:38:01.051110 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [CANDIDATE]: Term 1 election: Requested vote from peers bc8a1701a2714e19a226e79ac9cc1441 (127.0.98.61:45329), ed0c2bbd155948069eda58c553946de5 (127.0.98.60:43653)
I20260927 16:38:01.051311 22760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:01.052381 22760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2903f23e4b8598de35741f53018a in term 1.
I20260927 16:38:01.052799 22661 leader_election.cc:302] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2903f23e4b8598de35741f53018a, bc8a1701a2714e19a226e79ac9cc1441; no voters:
I20260927 16:38:01.053081 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:01.053339 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [term 1 LEADER]: Becoming Leader. State: Replica: 36ef2903f23e4b8598de35741f53018a, State: Running, Role: LEADER
I20260927 16:38:01.053725 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.056332 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ef2903f23e4b8598de35741f53018a. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.056470 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:01.056980 22904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:01.058090 22904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:01.064126 22760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2903f23e4b8598de35741f53018a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:01.065646 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: No bootstrap required, opened a new log
I20260927 16:38:01.070636 22898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:01.075474 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ef2903f23e4b8598de35741f53018a. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.075623 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:01.076025 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.076313 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0c2bbd155948069eda58c553946de5, State: Initialized, Role: FOLLOWER
I20260927 16:38:01.076665 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } }
I20260927 16:38:01.078125 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:01.078490 22836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:01.082610 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36ef2903f23e4b8598de35741f53018a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed0c2bbd155948069eda58c553946de5"
I20260927 16:38:01.083925 22836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2903f23e4b8598de35741f53018a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:01.084652 22898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:01.084875 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.059596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.0.98.60:43653)
0927 16:38:01.059722 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.059727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.067014 (+ 7287us) server_negotiation.cc:407] Connection header received
0927 16:38:01.067172 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.067175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.067220 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.067267 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.068166 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.068931 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.070485 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.070831 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.070906 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.081065 (+ 10159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.081070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.081073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.081103 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.082988 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.083518 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.083522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.083524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.083610 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.083900 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.083903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.083905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.084118 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.084229 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.084588 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.084712 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:38:01.086757 22835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2903f23e4b8598de35741f53018a in term 1.
I20260927 16:38:01.089632 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.089767 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:01.090925 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.091069 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:01.091475 22908 mvcc.cc:205] Tried to move back new op lower bound from 7333998924030726144 to 7333998924000473088. Current Snapshot: MvccSnapshot[applied={T|T < 7333998924030726144}]
I20260927 16:38:01.098728 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.059189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.98.61:45329)
0927 16:38:01.059315 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.059319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.064794 (+ 5475us) server_negotiation.cc:407] Connection header received
0927 16:38:01.064919 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.064924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.064977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.065051 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.066847 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.067611 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.077692 (+ 10081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.078052 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.078126 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.094638 (+ 16512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.094642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.094645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.094680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.096445 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.097241 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.097245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.097247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.097339 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.097647 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.097650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.097652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.097909 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.098021 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.098437 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.098573 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:38:01.099385 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.099552 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:01.100517 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ef2903f23e4b8598de35741f53018a. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.100651 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:01.103235 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.103384 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:01.103622 22909 mvcc.cc:205] Tried to move back new op lower bound from 7333998924030726144 to 7333998924000473088. Current Snapshot: MvccSnapshot[applied={T|T < 7333998924030726144}]
I20260927 16:38:01.104386 22924 mvcc.cc:205] Tried to move back new op lower bound from 7333998924030726144 to 7333998924000473088. Current Snapshot: MvccSnapshot[applied={T|T < 7333998924030726144}]
I20260927 16:38:01.104578 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ef2903f23e4b8598de35741f53018a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2903f23e4b8598de35741f53018a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } peers { permanent_uuid: "bc8a1701a2714e19a226e79ac9cc1441" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45329 } } peers { permanent_uuid: "ed0c2bbd155948069eda58c553946de5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43653 } } }
I20260927 16:38:01.104640 22904 catalog_manager.cc:1347] Generated new cluster ID: 9b4cc24b87dd4461b2641c5b33ce42e6
I20260927 16:38:01.104728 22904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:01.104724 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 16:38:01.106976 22926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:01.107041 22926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:01.107064 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:01.058622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.62:42787)
0927 16:38:01.058741 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.058745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.064456 (+ 5711us) server_negotiation.cc:407] Connection header received
0927 16:38:01.064953 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.064956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.065003 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.065056 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.068203 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.068951 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.075379 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:01.076061 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:01.076146 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:01.100432 (+ 24286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:01.100438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:01.100442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:01.100479 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:01.102475 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.105610 (+ 3135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.105615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.105617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.105708 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:01.106025 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:01.106029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:01.106031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:01.106274 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:01.106405 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:01.106769 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 16:38:01.106904 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:38:01.131973 22904 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:01.133347 22904 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:01.150449 22904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36ef2903f23e4b8598de35741f53018a: Generated new TSK 0
I20260927 16:38:01.151400 22904 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:38:01.186293 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.182781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.0.98.62:42787)
0927 16:38:01.182911 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.182917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.184500 (+ 1583us) server_negotiation.cc:407] Connection header received
0927 16:38:01.185025 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.185029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.185083 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.185142 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.186102 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35840: BlockingRecv error: recv got EOF from 127.0.0.1:35840 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 16:38:01.187148 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.182236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.98.61:45329)
0927 16:38:01.182410 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.182417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.184706 (+ 2289us) server_negotiation.cc:407] Connection header received
0927 16:38:01.185286 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.185289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.185351 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.185409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.187021 (+ 1612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51416: BlockingRecv error: recv got EOF from 127.0.0.1:51416 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 16:38:01.188832 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.183836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.0.98.60:43653)
0927 16:38:01.183966 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.183970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.187544 (+ 3574us) server_negotiation.cc:407] Connection header received
0927 16:38:01.187920 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.187924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.187983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.188036 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.188698 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40540: BlockingRecv error: recv got EOF from 127.0.0.1:40540 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 16:38:01.194057 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.192329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.0.98.61:45329)
0927 16:38:01.192460 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.192466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.193076 (+ 610us) server_negotiation.cc:407] Connection header received
0927 16:38:01.193318 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.193322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.193403 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.193456 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.193958 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51426: BlockingRecv error: recv got EOF from 127.0.0.1:51426 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 16:38:01.199393 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.197702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.0.98.61:45329)
0927 16:38:01.197807 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.197811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.198468 (+ 657us) server_negotiation.cc:407] Connection header received
0927 16:38:01.198646 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.198650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.198702 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.198755 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.199266 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51432: BlockingRecv error: recv got EOF from 127.0.0.1:51432 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:38:01.205055 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.203437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.0.98.61:45329)
0927 16:38:01.203542 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.203546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.204087 (+ 541us) server_negotiation.cc:407] Connection header received
0927 16:38:01.204284 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.204288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.204354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.204415 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.204950 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51436: BlockingRecv error: recv got EOF from 127.0.0.1:51436 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:38:01.213093 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.211309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.98.61:45329)
0927 16:38:01.211463 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.211468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.212061 (+ 593us) server_negotiation.cc:407] Connection header received
0927 16:38:01.212267 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.212271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.212327 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.212388 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.212974 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51440: BlockingRecv error: recv got EOF from 127.0.0.1:51440 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 16:38:01.225319 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.223562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.0.98.61:45329)
0927 16:38:01.223680 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.223684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.224279 (+ 595us) server_negotiation.cc:407] Connection header received
0927 16:38:01.224477 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.224480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.224532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.224583 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.225196 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51448: BlockingRecv error: recv got EOF from 127.0.0.1:51448 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:38:01.246037 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.244101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.0.98.61:45329)
0927 16:38:01.244230 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.244234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.244893 (+ 659us) server_negotiation.cc:407] Connection header received
0927 16:38:01.245151 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.245155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.245218 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.245279 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.245907 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51464: BlockingRecv error: recv got EOF from 127.0.0.1:51464 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:38:01.281980 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.280710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.0.98.61:45329)
0927 16:38:01.280838 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.280844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.281092 (+ 248us) server_negotiation.cc:407] Connection header received
0927 16:38:01.281253 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.281257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.281313 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.281408 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.281864 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:38:01.349416 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.348223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.0.98.61:45329)
0927 16:38:01.348453 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.348457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.348594 (+ 137us) server_negotiation.cc:407] Connection header received
0927 16:38:01.348688 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.348690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.348740 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.348806 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.349262 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51476: BlockingRecv error: recv got EOF from 127.0.0.1:51476 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260927 16:38:01.481076 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.479556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.0.98.61:45329)
0927 16:38:01.479734 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.479739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.479945 (+ 206us) server_negotiation.cc:407] Connection header received
0927 16:38:01.480060 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.480063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.480114 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.480186 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.480910 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51492: BlockingRecv error: recv got EOF from 127.0.0.1:51492 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 16:38:01.740391 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:01.739239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.98.61:45329)
0927 16:38:01.739395 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:01.739399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:01.739496 (+ 97us) server_negotiation.cc:407] Connection header received
0927 16:38:01.739646 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:01.739649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:01.739699 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:01.739789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:01.740238 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51504: BlockingRecv error: recv got EOF from 127.0.0.1:51504 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 16:38:02.036079 22895 catalog_manager.cc:1259] Loaded cluster ID: 9b4cc24b87dd4461b2641c5b33ce42e6
I20260927 16:38:02.036163 22895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: loading cluster ID for follower catalog manager: success
I20260927 16:38:02.037905 22895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: acquiring CA information for follower catalog manager: success
I20260927 16:38:02.039093 22895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc8a1701a2714e19a226e79ac9cc1441: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:38:02.108305 22926 catalog_manager.cc:1259] Loaded cluster ID: 9b4cc24b87dd4461b2641c5b33ce42e6
I20260927 16:38:02.108414 22926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: loading cluster ID for follower catalog manager: success
I20260927 16:38:02.110121 22926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: acquiring CA information for follower catalog manager: success
I20260927 16:38:02.111301 22926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed0c2bbd155948069eda58c553946de5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:38:02.258085 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:02.254507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.0.98.61:45329)
0927 16:38:02.254742 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:02.254745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:02.254762 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:38:02.254885 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:02.254888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:02.255037 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:02.255149 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:02.255384 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.255903 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.256719 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.256935 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.257025 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:02.257286 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:02.257736 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:02.257832 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:02.257883 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:38:02.257935 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260927 16:38:02.265148 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:02.261260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.0.98.60:43653)
0927 16:38:02.261511 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:02.261516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:02.261529 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:38:02.261681 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:02.261683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:02.261796 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:02.261917 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:02.262109 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.262680 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.263507 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.263739 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.263845 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:02.264143 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:02.264583 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:02.264892 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:02.264930 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 16:38:02.264988 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260927 16:38:02.272028 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:02.267956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.98.60:43653)
0927 16:38:02.268201 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:02.268206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:02.268476 (+ 270us) server_negotiation.cc:407] Connection header received
0927 16:38:02.268576 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:02.268579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:02.268623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:02.268692 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:02.268957 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.269552 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.270800 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.271168 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.271296 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:02.271563 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:02.271630 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:02.271736 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:02.271844 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 16:38:02.271894 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260927 16:38:02.272028 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:02.267911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.0.98.61:45329)
0927 16:38:02.268105 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:02.268111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:02.268507 (+ 396us) server_negotiation.cc:407] Connection header received
0927 16:38:02.268729 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:02.268733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:02.268786 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:02.268860 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:02.269103 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.269711 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.270882 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.271143 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.271260 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:02.271509 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:02.271587 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:02.271664 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:02.271814 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 16:38:02.271868 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 16:38:02.273274 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:02.267922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.0.98.62:42787)
0927 16:38:02.268338 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:02.268343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:02.268701 (+ 358us) server_negotiation.cc:407] Connection header received
0927 16:38:02.268795 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:02.268798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:02.268974 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:02.269104 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:02.270021 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.270611 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.271752 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:02.272088 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:02.272192 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:02.272616 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:02.272993 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:02.273064 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:02.273123 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:38:02.273169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":65,"threads_started":1}
I20260927 16:38:02.280921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22643
I20260927 16:38:02.287011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22718
I20260927 16:38:02.293020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 22793
2026-09-27T16:38:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3138 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:02 dist-test-slave-t27c krb5kdc[23013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:02 dist-test-slave-t27c krb5kdc[23013](info): set up 2 sockets
Sep 27 16:38:02 dist-test-slave-t27c krb5kdc[23013](info): commencing operation
krb5kdc: starting...
W20260927 16:38:03.330817 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:03.448905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:03.558929 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:03.559221 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:03.559244 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:03.559293 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:03.563230 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:03.563285 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:03.563304 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:03.563323 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:03.563338 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:03.568532 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:03.569545 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:03.570509 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:03.578886 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:03.578881 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:03.578902 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:03.579070 23026 server_base.cc:1034] running on GCE node
I20260927 16:38:03.579814 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:03.580965 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:03.582124 23026 hybrid_clock.cc:648] HybridClock initialized: now 1790527083582096 us; error 40 us; skew 500 ppm
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:03.584964 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:03.585810 23026 webserver.cc:460] Webserver started at http://127.0.98.62:44037/ using document root <none> and password file <none>
I20260927 16:38:03.586434 23026 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:03.586506 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:03.586673 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:03.588660 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data/instance:
uuid: "4020376daa094b189d4547adbfa1177e"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "1a9e3ef9bca6921232169cc37845c41b"
server_key_iv: "a21326555d74dda2a101328b4077c95d"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.589166 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance:
uuid: "4020376daa094b189d4547adbfa1177e"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "1a9e3ef9bca6921232169cc37845c41b"
server_key_iv: "a21326555d74dda2a101328b4077c95d"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.592705 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:38:03.595010 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:03.596151 23026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:03.596272 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal
uuid: "4020376daa094b189d4547adbfa1177e"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "1a9e3ef9bca6921232169cc37845c41b"
server_key_iv: "a21326555d74dda2a101328b4077c95d"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.596369 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:03.611910 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:03.615047 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:03.615252 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:03.621950 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36679
I20260927 16:38:03.622264 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36679 every 8 connection(s)
I20260927 16:38:03.622991 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:03.624766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 23026
I20260927 16:38:03.624857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:03.625140 392 external_mini_cluster.cc:1428] Setting key 30b414d3968cb838183cb6e9526fee31
I20260927 16:38:03.625813 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:03.630050 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:03.645227 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.633415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.98.62:36679)
0927 16:38:03.633835 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:03.633845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:03.633889 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:38:03.634397 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:03.634402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:03.634682 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:03.634858 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:03.635668 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.636607 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.637606 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.638000 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.638210 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.640791 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:03.640797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:03.640805 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:03.640826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:03.642474 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.642956 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.642961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.642962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.643062 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.643365 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.643368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.643369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.643747 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:03.643879 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:03.644217 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 16:38:03.644537 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":313,"thread_start_us":78,"threads_started":1}
I20260927 16:38:03.645236 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.632521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36679 (local address 127.0.0.1:41224)
0927 16:38:03.633076 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.633117 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:03.634049 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:03.634852 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:03.634857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:03.635282 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:03.635534 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.635547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.636744 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.636747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.637496 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.637505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.638141 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.638145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.638259 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.638819 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:03.638835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:03.640654 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:03.642575 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.642584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.642594 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.642855 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.643143 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.643145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.643147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.643288 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.643864 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:03.643868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:03.643991 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 16:38:03.644129 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:38:03.644430 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":4,"thread_start_us":164,"threads_started":1}
I20260927 16:38:03.647663 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:03.648372 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.648108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37312)
0927 16:38:03.648242 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.648314 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:38:03.648548 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
I20260927 16:38:03.649343 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.649149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37324)
0927 16:38:03.649251 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.649294 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 16:38:03.649469 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:38:03.650588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46343
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:38:03.698727 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } attempt: 1
I20260927 16:38:03.699653 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.699349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37338)
0927 16:38:03.699509 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.699577 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 16:38:03.700418 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.700199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37350)
0927 16:38:03.700305 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.700354 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:38:03.700549 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
W20260927 16:38:03.759783 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:03.760095 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:03.760159 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:03.760238 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:03.764206 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:03.764288 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:03.764325 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:03.764382 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:03.764418 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:03.768705 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } attempt: 2
I20260927 16:38:03.769502 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.769213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37366)
0927 16:38:03.769359 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.769429 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 16:38:03.769723 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:03.770215 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.769962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37380)
0927 16:38:03.770070 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.770123 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:38:03.770339 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46343: connect: Connection refused (error 111)
I20260927 16:38:03.770896 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:03.771878 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:03.780261 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:03.780261 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:03.780400 23101 server_base.cc:1034] running on GCE node
W20260927 16:38:03.780217 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:03.781180 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:03.782459 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:03.783638 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790527083783617 us; error 35 us; skew 500 ppm
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:03.786504 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:38:03.787346 23101 webserver.cc:460] Webserver started at http://127.0.98.61:46067/ using document root <none> and password file <none>
I20260927 16:38:03.787866 23101 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:03.787923 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:03.788086 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:03.790074 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data/instance:
uuid: "388cafc203844e2582bc252986b241dc"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "5db527257792702e906306c222068037"
server_key_iv: "5434b33cf9fdfd1d3a7c827132c1ff22"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.790635 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal/instance:
uuid: "388cafc203844e2582bc252986b241dc"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "5db527257792702e906306c222068037"
server_key_iv: "5434b33cf9fdfd1d3a7c827132c1ff22"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.794211 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:38:03.796583 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:03.797885 23101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:03.798051 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal
uuid: "388cafc203844e2582bc252986b241dc"
format_stamp: "Formatted at 2026-09-27 16:38:03 on dist-test-slave-t27c"
server_key: "5db527257792702e906306c222068037"
server_key_iv: "5434b33cf9fdfd1d3a7c827132c1ff22"
server_key_version: "encryptionkey@0"
I20260927 16:38:03.798162 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:03.827482 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:03.830312 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:03.830516 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:03.837368 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46343
I20260927 16:38:03.837657 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46343 every 8 connection(s)
I20260927 16:38:03.838369 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:38:03.841013 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:03.844596 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:03.846549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 23101
I20260927 16:38:03.846697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-1/wal/instance
I20260927 16:38:03.846984 392 external_mini_cluster.cc:1428] Setting key 779f0d0f5db85a04ba492ce8082caa1d
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 16:38:03.860008 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.846403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.0.98.62:36679)
0927 16:38:03.846578 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:03.846583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:03.847461 (+ 878us) server_negotiation.cc:407] Connection header received
0927 16:38:03.848427 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:03.848430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:03.848499 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:03.848579 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:03.850054 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.850981 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.852194 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.852629 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.852762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.855637 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:03.855640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:03.855643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:03.855671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:03.857239 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.857809 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.857812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.857814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.857904 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.858360 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.858366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.858367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.858627 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:03.858729 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:03.859579 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 16:38:03.859733 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:38:03.860975 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.846829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36679 (local address 127.0.0.1:41238)
0927 16:38:03.847299 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.847350 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:03.848176 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:03.848782 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:03.848793 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:03.849311 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:03.849894 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.849907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.851103 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.851105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.852065 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.852074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.852780 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.852782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.852922 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.853586 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:03.853602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:03.855437 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:03.857401 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.857407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.857422 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.857705 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.858026 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.858030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.858032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.858246 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.858774 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:03.858783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:03.859243 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 16:38:03.859478 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 16:38:03.859781 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260927 16:38:03.861490 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } attempt: 3
I20260927 16:38:03.862668 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:38:03 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:38:03.874838 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.861908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37396)
0927 16:38:03.862196 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.862212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:03.862285 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:03.863049 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:03.863053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:03.863267 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:03.863417 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.863428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.865148 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.865151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.865805 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.865814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.866715 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.866719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.866820 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.867360 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:03.867374 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:03.870767 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:03.873040 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.873059 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.873063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.873333 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.873645 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.873650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.873653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.873750 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.874286 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:03.874289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:03.874405 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 16:38:03.874499 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 16:38:03.874669 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260927 16:38:03.874871 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.862019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.61:46343)
0927 16:38:03.862555 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:03.862568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:03.862592 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:38:03.862638 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:03.862647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:03.862864 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:03.863600 (+ 736us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:03.863735 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.865013 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.865927 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.866608 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.866729 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.870897 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:03.870908 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:03.870925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:03.870951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:03.872891 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.873439 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.873444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.873445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.873524 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.873841 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.873846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.873847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.874201 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:03.874312 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:03.874611 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:38:03.874741 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":174,"threads_started":1}
I20260927 16:38:03.875905 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.863120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37412)
0927 16:38:03.863900 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.863912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:03.863981 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:03.865535 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:03.865538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:03.865706 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:03.865811 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.865817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.867065 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.867067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.867865 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:03.867878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.868313 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.868316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:03.868430 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.868901 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:03.868912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:03.871305 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:03.873782 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.873785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.873786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.874052 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.874371 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:03.874373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:03.874375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:03.874451 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:03.875396 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:03.875399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:03.875487 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 16:38:03.875593 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:38:03.875724 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":3}
I20260927 16:38:03.875993 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.863328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.61:46343)
0927 16:38:03.865298 (+ 1970us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:03.865307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:03.865318 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:38:03.865348 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:03.865351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:03.865398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:03.865500 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:03.866247 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.866929 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.867966 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:03.868212 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:03.868315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:03.871469 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:03.871472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:03.871474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:03.871492 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:03.873691 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.874152 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.874190 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.874195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.874282 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:03.874598 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:03.874601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:03.874602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:03.874824 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:03.874923 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:03.875649 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 16:38:03.875812 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":1889,"thread_start_us":108,"threads_started":1}
I20260927 16:38:03.876778 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:03.876797 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:03.877424 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.877201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38696)
0927 16:38:03.877303 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.877377 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 16:38:03.877424 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.877212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38704)
0927 16:38:03.877315 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.877364 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab.
W20260927 16:38:03.877825 23119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.878238 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.877913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38712)
0927 16:38:03.878082 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.878131 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 16:38:03.878343 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.878708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
/tmp/dist-test-taskxe3jhj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35699
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:38:03.878777 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.878552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38722)
0927 16:38:03.878664 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.878716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:38:03.878911 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.909566 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 1
I20260927 16:38:03.910490 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.910162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38724)
0927 16:38:03.910333 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.910398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:38:03.911187 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.910984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38734)
0927 16:38:03.911088 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.911136 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:38:03.911306 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.928148 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 1
I20260927 16:38:03.928939 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.928670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38748)
0927 16:38:03.928808 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.928866 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:38:03.929643 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.929423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38750)
0927 16:38:03.929549 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.929591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 16:38:03.929767 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.986454 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 2
I20260927 16:38:03.987249 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.986956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38764)
0927 16:38:03.987108 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.987175 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 16:38:03.987969 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.987765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38780)
0927 16:38:03.987870 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.987919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:38:03.988092 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
W20260927 16:38:03.989568 23177 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:38:03.989863 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:03.989918 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:03.989996 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:03.993986 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:03.994053 23177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:03.994108 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:03.994160 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:03.994216 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:03.997917 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 2
I20260927 16:38:03.998587 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.998337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38782)
0927 16:38:03.998452 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.998523 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 16:38:03.999233 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:03.999031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38796)
0927 16:38:03.999126 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:03.999170 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 16:38:03.999387 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35699: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35699: connect: Connection refused (error 111)
I20260927 16:38:03.999616 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36679,127.0.98.61:46343,127.0.98.60:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:04.000686 23177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:04.001708 23177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:04.010077 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:04.010077 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:04.010082 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:04.010310 23177 server_base.cc:1034] running on GCE node
I20260927 16:38:04.010839 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:04.011997 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:04.013170 23177 hybrid_clock.cc:648] HybridClock initialized: now 1790527084013153 us; error 27 us; skew 500 ppm
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:04.016117 23177 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:38:04.017002 23177 webserver.cc:460] Webserver started at http://127.0.98.60:43717/ using document root <none> and password file <none>
I20260927 16:38:04.017547 23177 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:04.017606 23177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:04.017772 23177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:04.019846 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data/instance:
uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef"
format_stamp: "Formatted at 2026-09-27 16:38:04 on dist-test-slave-t27c"
server_key: "ff1a70ca2e479b126a0ff10c9543b4df"
server_key_iv: "18ffbd71fd4b96822066d59683270246"
server_key_version: "encryptionkey@0"
I20260927 16:38:04.020377 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal/instance:
uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef"
format_stamp: "Formatted at 2026-09-27 16:38:04 on dist-test-slave-t27c"
server_key: "ff1a70ca2e479b126a0ff10c9543b4df"
server_key_iv: "18ffbd71fd4b96822066d59683270246"
server_key_version: "encryptionkey@0"
I20260927 16:38:04.024165 23177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:38:04.026665 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:04.027886 23177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:04.028019 23177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal
uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef"
format_stamp: "Formatted at 2026-09-27 16:38:04 on dist-test-slave-t27c"
server_key: "ff1a70ca2e479b126a0ff10c9543b4df"
server_key_iv: "18ffbd71fd4b96822066d59683270246"
server_key_version: "encryptionkey@0"
I20260927 16:38:04.028126 23177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:04.044977 23177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:04.047680 23177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:04.047876 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:04.054716 23177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35699
I20260927 16:38:04.054999 23244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35699 every 8 connection(s)
I20260927 16:38:04.055770 23177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:38:04.058789 23245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:04.062575 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:04.064940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu as pid 23177
I20260927 16:38:04.065065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxe3jhj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526802550915-392-0/minicluster-data/master-2/wal/instance
I20260927 16:38:04.065397 392 external_mini_cluster.cc:1428] Setting key d5305ae0046db1384025db26bf699ef5
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:38:04.076313 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.064966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.98.62:36679)
0927 16:38:04.065133 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.065138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.065357 (+ 219us) server_negotiation.cc:407] Connection header received
0927 16:38:04.066589 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.066592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.066649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.066745 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.067791 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.068292 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.069240 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.069474 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.069582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.072125 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.072130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.072133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.072163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.073878 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.074693 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.074697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.074699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.074789 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.075159 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.075162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.075164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.075398 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.075493 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.076008 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.076150 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 16:38:04.077057 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.064697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36679 (local address 127.0.0.1:41250)
0927 16:38:04.065193 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:04.065244 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:04.066387 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:04.066843 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:04.066857 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:04.067179 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:04.067652 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.067665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.068400 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.068407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.069123 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.069131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.069612 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.069616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.069789 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.070405 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:04.070426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:04.071940 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:04.073994 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.074001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.074008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.074573 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.074884 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.074888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.074891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.075059 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.075496 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:04.075505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:04.075746 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 16:38:04.075912 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:38:04.076226 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":3,"thread_start_us":149,"threads_started":1}
I20260927 16:38:04.077057 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.066059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.0.98.60:35699)
0927 16:38:04.066329 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.066342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.066574 (+ 232us) server_negotiation.cc:407] Connection header received
0927 16:38:04.066811 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.066815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.066989 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.067164 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.067935 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.068760 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.069584 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.069922 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.070028 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.072547 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.072557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.072574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.072598 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.074690 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.075160 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.075165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.075167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.075243 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.075516 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.075519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.075520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.075758 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.075940 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.076180 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.076306 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":176,"spinlock_wait_cycles":1920,"thread_start_us":84,"threads_started":1}
I20260927 16:38:04.078593 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:04.087237 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 3
I20260927 16:38:04.088200 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.079171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46343 (local address 127.0.0.1:37424)
0927 16:38:04.079478 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:04.079490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:04.079573 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:04.080158 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:04.080163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:04.080359 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:04.080490 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.080498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.081446 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.081449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.082268 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.082279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.082800 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.082803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.082876 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.083315 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:04.083325 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:04.084451 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:04.086231 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.086234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.086236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.086523 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.086811 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.086816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.086819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.086922 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.087341 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:04.087347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:04.087754 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 16:38:04.087884 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 16:38:04.088014 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260927 16:38:04.088636 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.079336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.0.98.61:46343)
0927 16:38:04.079772 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.079777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.079793 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:38:04.079838 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.079842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.079909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.080002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.080609 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.081328 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.082384 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.082661 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.082773 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.084587 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.084590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.084592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.084613 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.086088 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.086636 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.086639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.086640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.086714 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.087018 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.087021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.087022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.087235 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.087342 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.087962 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.088090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:38:04.089229 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.079945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.62:36679)
0927 16:38:04.080038 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.080041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.080051 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:38:04.080177 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.080179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.080209 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.080269 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.080930 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.081583 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.082736 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.083038 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.083115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.085399 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.085402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.085404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.085424 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.087072 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.087519 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.087527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.087528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.087612 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.088138 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.088142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.088144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.088431 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.088617 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.088959 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.089075 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":44}
I20260927 16:38:04.089246 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:38:04.090521 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } attempt: 3
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:38:04.099057 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.087632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38810)
0927 16:38:04.087740 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:04.087755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:04.087825 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:04.088068 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:04.088072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:04.088229 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:04.088391 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.088399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.089856 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.089860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.090683 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.090694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.092238 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.092241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.092326 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.092783 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:04.092796 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:04.095076 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:04.097328 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.097331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.097333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.097594 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.097904 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.097907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.097909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.098045 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.098501 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:04.098507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:04.098623 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 16:38:04.098707 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:38:04.098858 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260927 16:38:04.100097 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.087703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.0.98.60:35699)
0927 16:38:04.087813 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.087816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.087828 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:38:04.087902 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.087905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.087944 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.088343 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.088503 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.089719 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.090810 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.092121 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.095218 (+ 3097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.095322 (+ 104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.095325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.095330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.095353 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.097201 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.097707 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.097710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.097712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.097787 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.098159 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.098162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.098164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.098389 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.099664 (+ 1275us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.099737 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.099908 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":46}
I20260927 16:38:04.100699 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.089708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38826)
0927 16:38:04.089804 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:04.089818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:04.089879 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:04.090416 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:04.090419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:04.090572 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:04.090645 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.090651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.092155 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.092161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.092952 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.092964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.093581 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.093585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.093672 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.094190 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:04.094206 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:04.095684 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:04.098868 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.098873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.098875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.099126 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.099482 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.099485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.099486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.099570 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.100140 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:04.100145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:04.100237 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:38:04.100352 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 16:38:04.100508 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
Sep 27 16:38:04 dist-test-slave-t27c krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:38:04.101395 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.089848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.98.60:35699)
0927 16:38:04.090213 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.090221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.090233 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:38:04.090262 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.090265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.090300 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.090738 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.090816 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.092040 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.093081 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.093470 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.093904 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.096476 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.096480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.096483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.096507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.098736 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.099293 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.099296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.099297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.099383 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.099680 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.099683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.099686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.100024 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.100905 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.101031 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.101162 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":4352,"thread_start_us":164,"threads_started":1}
I20260927 16:38:04.103960 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.090373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.98.61:46343)
0927 16:38:04.090471 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.090475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.090544 (+ 69us) server_negotiation.cc:407] Connection header received
0927 16:38:04.090712 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.090716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.090776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.090826 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.091964 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.092644 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.093659 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.093988 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.094058 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.097259 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.097266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.097271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.097309 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.101043 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.102346 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.102351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.102352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.102436 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.102813 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.102818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.102822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.103055 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.103252 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.103635 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.103759 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:38:04.104511 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: Bootstrap starting.
I20260927 16:38:04.104633 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: Bootstrap starting.
I20260927 16:38:04.104952 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.090839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35699 (local address 127.0.0.1:38832)
0927 16:38:04.090956 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:04.090969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:04.091033 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:04.096108 (+ 5075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:04.096112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:04.096312 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:04.096467 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.096475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.097586 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.097589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.098464 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:04.098477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.099338 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.099341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:04.099467 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.099911 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:04.099925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:04.101374 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:04.103367 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.103370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.103372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.103615 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.103907 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:04.103910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:04.103912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:04.103998 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:04.104527 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:04.104531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:04.104635 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 16:38:04.104716 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:38:04.104833 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 16:38:04.105504 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.091050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.98.60:35699)
0927 16:38:04.095834 (+ 4784us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.095850 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.095869 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:38:04.095908 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.095911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.095955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.096061 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.096620 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.097459 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.098646 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.099141 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.099665 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.101513 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.101516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.101519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.101542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.103243 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.103727 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.103732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.103734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.103808 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.104114 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.104118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.104121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.104318 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.105228 (+ 910us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.105299 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.105389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3650,"spinlock_wait_cycles":16000,"thread_start_us":108,"threads_started":1}
I20260927 16:38:04.107453 23245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:04.108372 23245 log.cc:824] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:04.108939 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: Bootstrap starting.
I20260927 16:38:04.110736 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: No bootstrap required, opened a new log
I20260927 16:38:04.111758 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:04.112583 23169 log.cc:824] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:04.113344 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:38:04.113915 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:04.114737 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: No bootstrap required, opened a new log
I20260927 16:38:04.115670 23245 raft_consensus.cc:348] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.114648 23068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4020376daa094b189d4547adbfa1177e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254
I20260927 16:38:04.115935 23245 raft_consensus.cc:729] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502f8f1d934420e8cf21a1ff1e6b0ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:04.116266 23245 consensus_queue.cc:260] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.116367 23094 log.cc:824] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:04.116811 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.116984 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.117702 23245 sys_catalog.cc:564] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:04.118126 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.118415 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388cafc203844e2582bc252986b241dc, State: Initialized, Role: FOLLOWER
I20260927 16:38:04.118742 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.120065 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: No bootstrap required, opened a new log
I20260927 16:38:04.120333 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:04.123059 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.123296 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4020376daa094b189d4547adbfa1177e, State: Initialized, Role: FOLLOWER
I20260927 16:38:04.123620 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.121227 23177 master_runner.cc:186] Error getting master registration for 127.0.98.62:36679: OK, instance_id { permanent_uuid: "4020376daa094b189d4547adbfa1177e" instance_seqno: 1790527083620058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:38:04.124585 23177 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:38:04.124861 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:04.124946 23068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:38:04.125061 23068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 0 FOLLOWER]: Advancing to term 1
W20260927 16:38:04.125329 23279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:04.125403 23279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:04.126348 23068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
W20260927 16:38:04.127797 23267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:04.127918 23267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:04.125888 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:36679: OK, instance_id { permanent_uuid: "4020376daa094b189d4547adbfa1177e" instance_seqno: 1790527083620058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:38:04.129460 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:38:04.130115 23292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:04.130308 23292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:04.130653 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4020376daa094b189d4547adbfa1177e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "388cafc203844e2582bc252986b241dc"
I20260927 16:38:04.130875 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:04.132364 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4020376daa094b189d4547adbfa1177e in term 1.
I20260927 16:38:04.132808 23042 leader_election.cc:302] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388cafc203844e2582bc252986b241dc, 4020376daa094b189d4547adbfa1177e; no voters:
I20260927 16:38:04.132973 23219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4020376daa094b189d4547adbfa1177e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef"
I20260927 16:38:04.133205 23219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:04.134238 23219 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4020376daa094b189d4547adbfa1177e in term 1.
I20260927 16:38:04.134357 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.134490 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.127175 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.127266 23068 leader_election.cc:288] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [CANDIDATE]: Term 1 election: Requested vote from peers 388cafc203844e2582bc252986b241dc (127.0.98.61:46343), d502f8f1d934420e8cf21a1ff1e6b0ef (127.0.98.60:35699)
I20260927 16:38:04.138567 23294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:04.138787 23294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [term 1 LEADER]: Becoming Leader. State: Replica: 4020376daa094b189d4547adbfa1177e, State: Running, Role: LEADER
I20260927 16:38:04.139181 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.139972 23026 master_runner.cc:263] No leader master found from master 4020376daa094b189d4547adbfa1177e
I20260927 16:38:04.140026 23026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:38:04.139140 23294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } }
I20260927 16:38:04.149173 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.137438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.0.98.60:35699)
0927 16:38:04.137563 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.137568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.140299 (+ 2731us) server_negotiation.cc:407] Connection header received
0927 16:38:04.140441 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.140444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.140489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.140583 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.141391 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.141930 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.143056 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.143406 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.143492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.145434 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.145438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.145441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.145467 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.147257 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.147766 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.147770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.147772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.147868 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.148169 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.148172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.148174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.148386 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.148507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.148890 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.149012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:38:04.151048 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4020376daa094b189d4547adbfa1177e. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.151185 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:04.160027 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.138292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.98.62:36679)
0927 16:38:04.138407 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.138412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.150904 (+ 12492us) server_negotiation.cc:407] Connection header received
0927 16:38:04.151097 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.151101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.151150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.151208 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.152089 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.152944 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.154263 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.154605 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.154678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.156376 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.156379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.156382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.156407 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.158083 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.158605 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.158609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.158611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.158694 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.159027 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.159033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.159037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.159298 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.159411 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.159746 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.159873 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 16:38:04.166445 23304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:04.167161 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:04.138713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.98.61:46343)
0927 16:38:04.138837 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.138842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.158432 (+ 19590us) server_negotiation.cc:407] Connection header received
0927 16:38:04.158615 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.158619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.158675 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.158730 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.159802 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.160528 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.161538 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:04.161877 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:04.161968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:04.163651 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:04.163655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:04.163658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:04.163685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:04.165342 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.165848 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.165852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.165854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.165935 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:04.166238 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:04.166241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:04.166243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:04.166447 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:04.166545 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:04.166888 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 16:38:04.167002 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:38:04.167444 23304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:04.175346 23219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [term 1 FOLLOWER]: Refusing update from remote peer 4020376daa094b189d4547adbfa1177e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:04.176103 23301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:04.180980 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4020376daa094b189d4547adbfa1177e. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.181119 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.183717 23142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [term 1 FOLLOWER]: Refusing update from remote peer 4020376daa094b189d4547adbfa1177e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:04.184284 23301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [LEADER]: Connected to new peer: Peer: permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:04.188481 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4020376daa094b189d4547adbfa1177e. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.188614 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.194684 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.194821 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:04.195327 23305 mvcc.cc:205] Tried to move back new op lower bound from 7333998936774438912 to 7333998936650371072. Current Snapshot: MvccSnapshot[applied={T|T < 7333998936774438912}]
I20260927 16:38:04.195840 23306 mvcc.cc:205] Tried to move back new op lower bound from 7333998936774438912 to 7333998936650371072. Current Snapshot: MvccSnapshot[applied={T|T < 7333998936774438912}]
I20260927 16:38:04.195976 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.196071 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.197221 23304 catalog_manager.cc:1347] Generated new cluster ID: 14d4007910134a77bb7ebf3d9373932b
I20260927 16:38:04.197288 23304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:04.215014 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.215174 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:04.216763 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4020376daa094b189d4547adbfa1177e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4020376daa094b189d4547adbfa1177e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36679 } } peers { permanent_uuid: "388cafc203844e2582bc252986b241dc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46343 } } peers { permanent_uuid: "d502f8f1d934420e8cf21a1ff1e6b0ef" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35699 } } }
I20260927 16:38:04.216897 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:38:04.217921 23308 mvcc.cc:205] Tried to move back new op lower bound from 7333998936774438912 to 7333998936650371072. Current Snapshot: MvccSnapshot[applied={T|T < 7333998936774438912}]
I20260927 16:38:04.219897 23304 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:04.221369 23304 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:04.243868 23304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4020376daa094b189d4547adbfa1177e: Generated new TSK 0
I20260927 16:38:04.244858 23304 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:38:04.281219 23251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.278337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.98.60:35699)
0927 16:38:04.278492 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.278497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.279773 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 16:38:04.280265 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.280269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.280325 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.280384 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.281080 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38858: BlockingRecv error: recv got EOF from 127.0.0.1:38858 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:38:04.282593 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.279076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.98.62:36679)
0927 16:38:04.279194 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.279198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.281701 (+ 2503us) server_negotiation.cc:407] Connection header received
0927 16:38:04.281861 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.281865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.281918 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.281967 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.282478 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41276: BlockingRecv error: recv got EOF from 127.0.0.1:41276 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:38:04.284034 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.279383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.0.98.61:46343)
0927 16:38:04.279511 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.279517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.283063 (+ 3546us) server_negotiation.cc:407] Connection header received
0927 16:38:04.283328 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.283332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.283385 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.283437 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.283918 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37452: BlockingRecv error: recv got EOF from 127.0.0.1:37452 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:38:04.288599 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.287066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.0.98.61:46343)
0927 16:38:04.287172 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.287177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.287666 (+ 489us) server_negotiation.cc:407] Connection header received
0927 16:38:04.287889 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.287892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.287967 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.288019 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.288496 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37460: BlockingRecv error: recv got EOF from 127.0.0.1:37460 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:38:04.293668 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.292266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.98.61:46343)
0927 16:38:04.292365 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.292369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.292814 (+ 445us) server_negotiation.cc:407] Connection header received
0927 16:38:04.292987 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.292990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.293036 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.293086 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.293572 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37462: BlockingRecv error: recv got EOF from 127.0.0.1:37462 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 16:38:04.299885 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.298255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.61:46343)
0927 16:38:04.298361 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.298365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.298846 (+ 481us) server_negotiation.cc:407] Connection header received
0927 16:38:04.299045 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.299049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.299103 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.299156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.299740 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37478: BlockingRecv error: recv got EOF from 127.0.0.1:37478 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:38:04.307941 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.306301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.98.61:46343)
0927 16:38:04.306441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.306446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.306956 (+ 510us) server_negotiation.cc:407] Connection header received
0927 16:38:04.307163 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.307166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.307221 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.307278 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.307832 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37494: BlockingRecv error: recv got EOF from 127.0.0.1:37494 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 16:38:04.319993 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.318418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.61:46343)
0927 16:38:04.318527 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.318531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.319032 (+ 501us) server_negotiation.cc:407] Connection header received
0927 16:38:04.319219 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.319223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.319272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.319338 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.319880 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58492: BlockingRecv error: recv got EOF from 127.0.0.1:58492 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:38:04.340364 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.338692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.0.98.61:46343)
0927 16:38:04.338822 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.338827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.339382 (+ 555us) server_negotiation.cc:407] Connection header received
0927 16:38:04.339578 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.339582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.339634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.339685 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.340244 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58504: BlockingRecv error: recv got EOF from 127.0.0.1:58504 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 16:38:04.376223 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.375070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.0.98.61:46343)
0927 16:38:04.375226 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.375231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.375352 (+ 121us) server_negotiation.cc:407] Connection header received
0927 16:38:04.375480 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.375484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.375543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.375647 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.376122 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58508: BlockingRecv error: recv got EOF from 127.0.0.1:58508 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 16:38:04.444041 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.442851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.0.98.61:46343)
0927 16:38:04.443012 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.443018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.443172 (+ 154us) server_negotiation.cc:407] Connection header received
0927 16:38:04.443338 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.443342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.443398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.443483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.443919 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58524: BlockingRecv error: recv got EOF from 127.0.0.1:58524 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 16:38:04.575536 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.574247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.0.98.61:46343)
0927 16:38:04.574411 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.574416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.574608 (+ 192us) server_negotiation.cc:407] Connection header received
0927 16:38:04.574747 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.574750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.574801 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.574880 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.575371 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58528: BlockingRecv error: recv got EOF from 127.0.0.1:58528 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 16:38:04.834858 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:04.833640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.0.98.61:46343)
0927 16:38:04.833779 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:04.833785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:04.833889 (+ 104us) server_negotiation.cc:407] Connection header received
0927 16:38:04.834078 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:04.834081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:04.834138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:04.834251 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:04.834736 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58530: BlockingRecv error: recv got EOF from 127.0.0.1:58530 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:38:05.126858 23279 catalog_manager.cc:1259] Loaded cluster ID: 14d4007910134a77bb7ebf3d9373932b
I20260927 16:38:05.126965 23279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: loading cluster ID for follower catalog manager: success
I20260927 16:38:05.128965 23279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: acquiring CA information for follower catalog manager: success
I20260927 16:38:05.129210 23267 catalog_manager.cc:1259] Loaded cluster ID: 14d4007910134a77bb7ebf3d9373932b
I20260927 16:38:05.129276 23267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: loading cluster ID for follower catalog manager: success
I20260927 16:38:05.130533 23279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 388cafc203844e2582bc252986b241dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:38:05.131189 23267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: acquiring CA information for follower catalog manager: success
I20260927 16:38:05.132340 23267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d502f8f1d934420e8cf21a1ff1e6b0ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:38:05.352566 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:05.348848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.0.98.61:46343)
0927 16:38:05.349121 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:05.349124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:05.349140 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:38:05.349262 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:05.349264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:05.349386 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:05.349497 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:05.349718 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.350246 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.351225 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.351442 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.351515 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:05.351804 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:05.352220 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:05.352306 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:05.352394 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 16:38:05.352446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260927 16:38:05.358872 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:05.355660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.0.98.60:35699)
0927 16:38:05.355883 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:05.355887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:05.355899 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:38:05.355983 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:05.355985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:05.356094 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:05.356189 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:05.356357 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.356856 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.357606 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.357820 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.357894 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:05.358135 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:05.358564 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:05.358650 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:05.358709 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:38:05.358753 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260927 16:38:05.365530 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:05.361790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.0.98.60:35699)
0927 16:38:05.361932 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:05.361936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:05.362093 (+ 157us) server_negotiation.cc:407] Connection header received
0927 16:38:05.362282 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:05.362288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:05.362344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:05.362391 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:05.362733 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.363365 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.364370 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.364726 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.364829 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:05.365053 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:05.365127 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:05.365190 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:05.365336 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 16:38:05.365399 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:38:05.366042 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:05.362046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.0.98.61:46343)
0927 16:38:05.362189 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:05.362193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:05.362406 (+ 213us) server_negotiation.cc:407] Connection header received
0927 16:38:05.362609 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:05.362612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:05.362674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:05.362750 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:05.363022 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.363777 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.364733 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.365084 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.365185 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:05.365466 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:05.365539 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:05.365590 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:05.365850 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 16:38:05.365902 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:38:05.366731 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:05.361746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.0.98.62:36679)
0927 16:38:05.361995 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:05.362001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:05.362332 (+ 331us) server_negotiation.cc:407] Connection header received
0927 16:38:05.362471 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:05.362474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:05.362639 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:05.362763 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:05.363113 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.363902 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.364957 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:05.365321 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:05.365411 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:05.365821 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:05.366203 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:05.366290 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:05.366535 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 16:38:05.366595 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260927 16:38:05.369427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 23026
I20260927 16:38:05.376142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 23101
I20260927 16:38:05.382509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxe3jhj/build/debug/bin/kudu with pid 23177
2026-09-27T16:38:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3089 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6228 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282824 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST